Jul 14 22:08:54.055507 ip-10-0-131-45 systemd[1]: Starting Kubernetes Kubelet... Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.518049 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524253 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524268 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524272 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524275 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524278 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524281 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524284 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:08:54.637656 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524287 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524291 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524294 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524296 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524299 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524302 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524305 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524307 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524310 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524313 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524316 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524326 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524329 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524332 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524334 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524337 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524340 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524342 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524345 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524348 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:08:54.652058 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524351 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:08:54.596270 ip-10-0-131-45 systemd[1]: Started Kubernetes Kubelet. Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524353 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524356 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524359 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524362 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524365 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524367 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524370 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524373 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524375 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524378 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524381 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524383 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524386 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524389 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524392 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524395 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524398 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524401 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:08:54.681716 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524405 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524410 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524414 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524417 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524420 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524423 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524425 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524428 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524430 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524433 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524436 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524438 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524441 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524443 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524446 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524448 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524451 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524454 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524456 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524459 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:08:54.818669 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524462 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524464 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524466 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524469 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524472 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524474 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524477 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524482 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524486 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524490 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524493 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524496 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524498 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524501 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524505 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524507 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524510 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524512 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524515 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:08:54.820672 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524518 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524520 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524523 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524526 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524528 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524531 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524534 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524536 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524539 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524541 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524544 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524546 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524549 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.524552 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526409 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526417 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526421 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526424 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526427 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526432 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:08:54.822797 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526436 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526439 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526442 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526445 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526448 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526450 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526453 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526456 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526460 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526462 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526465 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526467 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526470 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526472 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526475 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526478 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526481 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526484 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526487 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:08:54.824564 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526490 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526492 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526495 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526497 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526500 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526503 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526506 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526508 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526511 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526513 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526516 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526518 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526521 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526525 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526529 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526532 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526535 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526537 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526540 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:08:54.826870 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526542 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526545 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526547 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526550 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526553 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526555 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526558 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526560 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526563 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526566 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526568 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526571 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526574 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526576 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526579 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526582 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526584 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526587 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526589 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526592 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:08:54.828794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526594 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526597 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526599 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526602 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526608 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526611 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526614 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526618 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526621 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526623 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526626 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526629 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526631 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526635 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526637 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526640 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526643 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526646 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526648 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:08:54.832894 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526651 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526654 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526656 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526659 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526661 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526664 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526666 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526668 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526671 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526674 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526676 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526679 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526682 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526684 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526687 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.526690 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527065 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527070 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527074 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527077 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:08:54.834880 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527080 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527083 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527086 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527088 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527090 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527096 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527099 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527101 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527104 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527107 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527110 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527113 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527115 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527118 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527121 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527124 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527128 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527131 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527133 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:08:54.837027 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527136 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527139 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527141 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527144 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527147 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527150 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527153 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527156 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527158 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527161 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527164 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527166 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527169 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527172 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527174 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527177 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527179 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527182 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527186 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:08:54.839172 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527189 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527191 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527194 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527196 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527199 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527202 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527204 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527207 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527209 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527212 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527215 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527217 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527220 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527222 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527225 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527228 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527230 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527232 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527253 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527256 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:08:54.840198 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527259 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527261 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527264 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527267 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527269 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527272 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527274 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527277 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527280 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527282 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527284 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527290 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527292 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527295 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527297 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527300 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527302 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527307 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527310 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527313 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:08:54.841727 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527316 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527320 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527323 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527326 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527328 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527332 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527335 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527339 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527341 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527344 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527348 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527351 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527353 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527356 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527358 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527361 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.527363 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527426 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527433 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527440 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 22:08:54.843563 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527445 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527449 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527452 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527458 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527461 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527465 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527468 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527471 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527475 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527478 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527481 2578 flags.go:64] FLAG: --cgroup-root="" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527484 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527487 2578 flags.go:64] FLAG: --client-ca-file="" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527490 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527493 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527497 2578 flags.go:64] FLAG: --cluster-domain="" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527500 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527503 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527506 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527509 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527512 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527515 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527519 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 22:08:54.844463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527523 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527538 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527542 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527545 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527549 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527552 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527556 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527559 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527562 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527565 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527568 2578 flags.go:64] FLAG: --enable-server="true" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527571 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527577 2578 flags.go:64] FLAG: --event-burst="100" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527580 2578 flags.go:64] FLAG: --event-qps="50" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527583 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527587 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527590 2578 flags.go:64] FLAG: --eviction-hard="" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527593 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527596 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527599 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527604 2578 flags.go:64] FLAG: --eviction-soft="" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527607 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527609 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527612 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527615 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 22:08:54.845489 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527618 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527621 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527623 2578 flags.go:64] FLAG: --feature-gates="" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527627 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527630 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527633 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527637 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527640 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527643 2578 flags.go:64] FLAG: --help="false" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527646 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-45.ec2.internal" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527649 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527652 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527655 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527658 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527661 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527664 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527666 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527669 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527672 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527676 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527679 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527682 2578 flags.go:64] FLAG: --kube-reserved="" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527685 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527687 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 22:08:54.846600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527690 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527693 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527696 2578 flags.go:64] FLAG: --lock-file="" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527699 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527702 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527705 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527711 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527713 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527716 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527719 2578 flags.go:64] FLAG: --logging-format="text" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527722 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527725 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527728 2578 flags.go:64] FLAG: --manifest-url="" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527731 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527736 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527739 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527743 2578 flags.go:64] FLAG: --max-pods="110" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527745 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527748 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527751 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527754 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527757 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527760 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527762 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527769 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 22:08:54.847705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527772 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527775 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527779 2578 flags.go:64] FLAG: --pod-cidr="" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527782 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527787 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527790 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527793 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527796 2578 flags.go:64] FLAG: --port="10250" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527799 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527802 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a092102ea19a7e44" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527805 2578 flags.go:64] FLAG: --qos-reserved="" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527808 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527812 2578 flags.go:64] FLAG: --register-node="true" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527815 2578 flags.go:64] FLAG: --register-with-taints="" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527819 2578 flags.go:64] FLAG: --registry-burst="10" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527821 2578 flags.go:64] FLAG: --registry-qps="5" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527824 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527827 2578 flags.go:64] FLAG: --reserved-memory="" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527830 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527833 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527836 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527839 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527843 2578 flags.go:64] FLAG: --runonce="false" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527846 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527849 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 22:08:54.851946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527852 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527855 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527857 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527860 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527863 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527866 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527869 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527872 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527875 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527879 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527882 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527885 2578 flags.go:64] FLAG: --system-cgroups="" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527888 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527893 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527895 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527898 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527916 2578 flags.go:64] FLAG: --tls-min-version="" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527920 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527923 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527926 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527930 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527933 2578 flags.go:64] FLAG: --v="2" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527937 2578 flags.go:64] FLAG: --version="false" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527941 2578 flags.go:64] FLAG: --vmodule="" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527944 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 22:08:54.853018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.527948 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528054 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528058 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528061 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528065 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528068 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528071 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528074 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528077 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528079 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528082 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528084 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528087 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528090 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528092 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528095 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528098 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528101 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528103 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528106 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:08:54.856479 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528108 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528111 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528114 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528116 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528120 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528123 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528126 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528128 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528132 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528134 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528137 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528139 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528142 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528144 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528147 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528150 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528153 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528156 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528158 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:08:54.857046 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528161 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528163 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528166 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528169 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528171 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528174 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528176 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528178 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528181 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528188 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528191 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528193 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528196 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528199 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528201 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528203 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528206 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528208 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528211 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528213 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:08:54.857575 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528216 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528218 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528221 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528224 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528226 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528229 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528231 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528234 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528251 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528258 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528262 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528266 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528269 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528272 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528275 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528278 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528282 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528286 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528289 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:08:54.927081 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528292 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528295 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528300 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528303 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528305 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528309 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528313 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528316 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528319 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528321 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528324 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528326 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528329 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528331 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528334 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528336 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528339 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528342 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528345 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528347 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:08:54.928120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528350 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.528353 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.528360 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.535307 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.535322 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.535336 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.535345 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535400 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535406 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535409 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535413 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535416 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535419 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535422 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:08:54.929211 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535425 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535427 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535430 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535432 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535437 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535440 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535444 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535447 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535451 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535455 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535458 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535461 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535464 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535467 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535469 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535472 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535475 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535481 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535483 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:08:54.929877 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535486 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535488 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535491 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535495 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535498 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535500 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535503 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535505 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535508 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535511 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535514 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535516 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535519 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535522 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535524 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535527 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535530 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535532 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535535 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535537 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:08:54.930768 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535540 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535542 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535545 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535548 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535550 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535553 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535556 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535558 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535561 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535563 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535567 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535572 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535574 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535577 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535579 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535582 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535586 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535588 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535591 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535593 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:08:54.931655 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535596 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535598 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535601 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535605 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535608 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535612 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535614 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535617 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535620 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535623 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535626 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535629 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535632 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535635 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535637 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535640 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535642 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535645 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535648 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535650 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:08:54.932545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535653 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535655 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535659 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535666 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535668 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535671 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535673 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535676 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535680 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535682 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535685 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535688 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535690 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.535695 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535802 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:08:54.933444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535806 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535809 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535812 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535815 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535818 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535821 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535823 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535825 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535828 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535830 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535833 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535836 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535838 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535841 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535845 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535849 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535852 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535854 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535857 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:08:54.934120 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535861 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535867 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535870 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535872 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535876 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535879 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535882 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535884 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535887 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535890 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535893 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535895 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535898 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535900 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535903 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535905 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535908 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535910 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535913 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535915 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:08:54.934998 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535918 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535920 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535923 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535925 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535927 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535930 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535933 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535935 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535937 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535940 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535942 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535945 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535947 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535952 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535955 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535957 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535960 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535962 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535965 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535967 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:08:54.935887 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535970 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535973 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535975 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535977 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535980 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535982 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535985 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535987 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535990 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535992 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535995 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.535998 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536000 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536002 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536005 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536007 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536010 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536013 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536015 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536018 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:08:54.936760 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536021 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536023 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536025 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536028 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536030 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536035 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536038 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536040 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536043 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536045 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536048 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536050 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536054 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536058 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536061 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536064 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536066 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536069 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:08:54.937663 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:54.536072 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.536077 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.536172 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.539504 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.541215 2578 server.go:1019] "Starting client certificate rotation" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.541265 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.541295 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.544082 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.548810 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.559193 2578 log.go:25] "Validated CRI v1 runtime API" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.561871 2578 log.go:25] "Validated CRI v1 image API" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.563797 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 22:08:54.938549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.566398 2578 fs.go:135] Filesystem UUIDs: map[0b671828-2a5a-4c80-99ad-93c9d1d16454:/dev/nvme0n1p3 2bc2feb7-337d-43f2-a05d-772458810001:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 14 22:08:54.939203 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.566418 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 22:08:54.939203 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.572328 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.575454 2578 manager.go:216] Machine: {Timestamp:2026-07-14 22:08:54.574015171 +0000 UTC m=+0.386646802 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098472 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23bf8ffc3594d4681f65997987a09c SystemUUID:ec23bf8f-fc35-94d4-681f-65997987a09c BootID:cea27a4e-958a-49fa-b45e-7d4a2e0ce289 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:80:a5:d7:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:80:a5:d7:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:16:d5:32:75:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.575554 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.575625 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.579087 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.579110 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.579922 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.579932 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.579940 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.579959 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.581234 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.586045 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.586175 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.586206 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.586216 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.586234 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.586260 2578 kubelet.go:398] "Adding apiserver pod source" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.586272 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 22:08:54.939628 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.586683 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.586705 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.589026 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.591599 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.591626 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592405 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592421 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592429 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592436 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592445 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592453 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592484 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592492 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592499 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592505 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592515 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592523 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592551 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.592556 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.595447 2578 server.go:1268] "Started kubelet" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.595539 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.595585 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.596306 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.597649 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.598146 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.598281 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.598318 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.600266 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.602313 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.602223 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-45.ec2.internal.18c2476cb4148633 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-45.ec2.internal,UID:ip-10-0-131-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-45.ec2.internal,},FirstTimestamp:2026-07-14 22:08:54.595421747 +0000 UTC m=+0.408053379,LastTimestamp:2026-07-14 22:08:54.595421747 +0000 UTC m=+0.408053379,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-45.ec2.internal,}" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.607765 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.607781 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.607847 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608445 2578 factory.go:55] Registering systemd factory Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608459 2578 factory.go:223] Registration of the systemd container factory successfully Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608534 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608538 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608550 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608653 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608664 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608730 2578 factory.go:153] Registering CRI-O factory Jul 14 22:08:54.942802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608745 2578 factory.go:223] Registration of the crio container factory successfully Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608791 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608813 2578 factory.go:103] Registering Raw factory Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.608828 2578 manager.go:1196] Started watching for new ooms in manager Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.609219 2578 manager.go:349] Starting recovery of all containers Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.610084 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.610157 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.616443 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.617583 2578 manager.go:354] Recovery completed Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.620791 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jgqbf" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.622925 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.625256 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.625278 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.625287 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.625788 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.625798 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.625808 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.625816 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.626449 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.626470 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.626480 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.627980 2578 policy_none.go:49] "None policy: Start" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.627992 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.628001 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.628507 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jgqbf" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.629954 2578 policy_none.go:47] "Start" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.687777 2578 manager.go:337] "Starting Device Plugin manager" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.687967 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.687978 2578 server.go:87] "Starting device plugin registration server" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.688184 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.688193 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.688342 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.688425 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.688432 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.688895 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.688949 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.703476 2578 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2618 into the system container \"/system.slice\": " Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.788433 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.789419 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.789453 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.789470 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.789498 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-45.ec2.internal" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.799080 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-45.ec2.internal" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.799098 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-45.ec2.internal\": node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:54.944531 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.829480 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.830548 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.830587 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.830608 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.830645 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.832927 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.929744 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.930829 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal"] Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.930900 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.931932 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.931966 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.931989 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.934321 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.934497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.934529 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.935073 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.935069 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.935102 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.935102 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.935117 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.935125 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.937379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.937404 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.938232 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.938270 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:08:54.952208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:54.938286 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.966768 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-45.ec2.internal\" not found" node="ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:54.970191 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-45.ec2.internal\" not found" node="ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.011423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e71ccf9865e4302d818dc5edb3e82267-config\") pod \"kube-apiserver-proxy-ip-10-0-131-45.ec2.internal\" (UID: \"e71ccf9865e4302d818dc5edb3e82267\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.011445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04750a8bbdc612023d3422ff4bdfea8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"04750a8bbdc612023d3422ff4bdfea8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.011462 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04750a8bbdc612023d3422ff4bdfea8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"04750a8bbdc612023d3422ff4bdfea8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:55.030572 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.112231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e71ccf9865e4302d818dc5edb3e82267-config\") pod \"kube-apiserver-proxy-ip-10-0-131-45.ec2.internal\" (UID: \"e71ccf9865e4302d818dc5edb3e82267\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.112272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04750a8bbdc612023d3422ff4bdfea8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"04750a8bbdc612023d3422ff4bdfea8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.112296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04750a8bbdc612023d3422ff4bdfea8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"04750a8bbdc612023d3422ff4bdfea8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.112337 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e71ccf9865e4302d818dc5edb3e82267-config\") pod \"kube-apiserver-proxy-ip-10-0-131-45.ec2.internal\" (UID: \"e71ccf9865e4302d818dc5edb3e82267\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.112343 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04750a8bbdc612023d3422ff4bdfea8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"04750a8bbdc612023d3422ff4bdfea8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.112344 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04750a8bbdc612023d3422ff4bdfea8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal\" (UID: \"04750a8bbdc612023d3422ff4bdfea8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.167995 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:55.131348 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:55.369793 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:55.231864 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:55.369793 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.269035 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.369793 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.274206 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.369793 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:55.332843 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:55.575436 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:55.433366 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:55.575436 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:55.534272 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-45.ec2.internal\" not found" Jul 14 22:08:55.575436 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.540457 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 22:08:55.575436 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.540607 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.599179 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.608997 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.609011 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.621313 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.626617 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.626693 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.634746 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 22:03:55 +0000 UTC" deadline="2028-01-26 17:25:54.344236556 +0000 UTC" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.634767 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13459h16m58.709472203s" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.641256 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.652363 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sx4rz" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.658433 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sx4rz" Jul 14 22:08:55.763115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.707966 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:08:55.966032 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:55.948799 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04750a8bbdc612023d3422ff4bdfea8f.slice/crio-106e856f881b7d75cf9a310ab365c137b38eb7b3d1fc6a2598b71d938f2a88a9 WatchSource:0}: Error finding container 106e856f881b7d75cf9a310ab365c137b38eb7b3d1fc6a2598b71d938f2a88a9: Status 404 returned error can't find the container with id 106e856f881b7d75cf9a310ab365c137b38eb7b3d1fc6a2598b71d938f2a88a9 Jul 14 22:08:55.966032 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:55.949181 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71ccf9865e4302d818dc5edb3e82267.slice/crio-ba3f90932c8823378169858ab839b78d0ad37298ed3cf800ffef60dec5df602b WatchSource:0}: Error finding container ba3f90932c8823378169858ab839b78d0ad37298ed3cf800ffef60dec5df602b: Status 404 returned error can't find the container with id ba3f90932c8823378169858ab839b78d0ad37298ed3cf800ffef60dec5df602b Jul 14 22:08:55.966032 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:55.956510 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:08:56.185090 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.106422 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:08:56.497262 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.430177 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.587299 2578 apiserver.go:52] "Watching apiserver" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.601108 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.601537 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-wkv7z","openshift-cluster-node-tuning-operator/tuned-lt2gz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal","openshift-multus/multus-additional-cni-plugins-d6zkw","openshift-network-diagnostics/network-check-target-fsbc5","openshift-network-operator/iptables-alerter-dtbcc","openshift-ovn-kubernetes/ovnkube-node-dsgzj","kube-system/global-pull-secret-syncer-f2j5k","kube-system/konnectivity-agent-v8ln8","kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74","openshift-dns/node-resolver-klpfw","openshift-image-registry/node-ca-lrct7","openshift-multus/multus-v9hsx"] Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.604824 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.607465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.610006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.612436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.614907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.614991 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.618523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.619772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-var-lib-cni-multus\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.619800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-node-log\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.619998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmhf\" (UniqueName: \"kubernetes.io/projected/05ccb78a-325c-4951-b04d-d4b614127b78-kube-api-access-rsmhf\") pod \"node-ca-lrct7\" (UID: \"05ccb78a-325c-4951-b04d-d4b614127b78\") " pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-var-lib-cni-bin\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620067 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-etc-kubernetes\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-var-lib-openvswitch\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05ccb78a-325c-4951-b04d-d4b614127b78-host\") pod \"node-ca-lrct7\" (UID: \"05ccb78a-325c-4951-b04d-d4b614127b78\") " pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-cni-dir\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-run-k8s-cni-cncf-io\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.654822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-run-netns\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-run-openvswitch\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-log-socket\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bhwq\" (UniqueName: \"kubernetes.io/projected/2519ea9e-2e8c-42a1-81c8-3649662d1c09-kube-api-access-4bhwq\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-systemd-units\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-cni-bin\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77c87b0b-9613-4070-a8e6-eea000f99c1a-ovnkube-config\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzqxt\" (UniqueName: \"kubernetes.io/projected/77c87b0b-9613-4070-a8e6-eea000f99c1a-kube-api-access-bzqxt\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.620987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-cnibin\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2519ea9e-2e8c-42a1-81c8-3649662d1c09-cni-binary-copy\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-run-netns\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-var-lib-kubelet\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-slash\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-run-ovn\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-system-cni-dir\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-os-release\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.656758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-kubelet\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-cni-netd\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-conf-dir\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-daemon-config\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-run-multus-certs\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-etc-openvswitch\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77c87b0b-9613-4070-a8e6-eea000f99c1a-env-overrides\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77c87b0b-9613-4070-a8e6-eea000f99c1a-ovnkube-script-lib\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/05ccb78a-325c-4951-b04d-d4b614127b78-serviceca\") pod \"node-ca-lrct7\" (UID: \"05ccb78a-325c-4951-b04d-d4b614127b78\") " pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-socket-dir-parent\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-hostroot\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-run-systemd\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.621733 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77c87b0b-9613-4070-a8e6-eea000f99c1a-ovn-node-metrics-cert\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.622130 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.622172 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.622211 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2rqrb\"" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.622414 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.622450 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.622496 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.622620 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 22:08:56.657967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.624540 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.624615 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.624951 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.624973 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lkqr6\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.625161 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.625186 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.626678 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j6txh\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.626853 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.626920 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j89dg\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.627005 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9f8qp\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.627118 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.627188 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.627336 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.627467 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sfn4f\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.627596 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.627859 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.627993 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.628128 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.628271 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.628449 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.628908 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.628965 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.629399 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.629473 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.631865 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.631999 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.631868 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ggb8x\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.632396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.637652 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.640055 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.640067 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.640401 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.641619 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vn8qq\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.641936 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.642311 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 22:08:56.659889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.643744 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-59qb8\"" Jul 14 22:08:56.661962 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.661509 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 22:03:56 +0000 UTC" deadline="2028-01-03 04:44:43.745142934 +0000 UTC" Jul 14 22:08:56.661962 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.661554 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12894h35m47.08359306s" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.713265 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.722844 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/514eec71-0eb2-4101-be06-0abbad141842-dbus\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.722879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.722907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-cnibin\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.722943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/05ccb78a-325c-4951-b04d-d4b614127b78-serviceca\") pod \"node-ca-lrct7\" (UID: \"05ccb78a-325c-4951-b04d-d4b614127b78\") " pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.722996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-socket-dir-parent\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-hostroot\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-sysctl-conf\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-sys\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-system-cni-dir\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-os-release\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d803a3ed-0104-4caf-a879-2d5d851f31e6-iptables-alerter-script\") pod \"iptables-alerter-dtbcc\" (UID: \"d803a3ed-0104-4caf-a879-2d5d851f31e6\") " pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-var-lib-cni-multus\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce9fa062-a9b3-4a71-a5c4-f76f22f36308-konnectivity-ca\") pod \"konnectivity-agent-v8ln8\" (UID: \"ce9fa062-a9b3-4a71-a5c4-f76f22f36308\") " pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723362 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-var-lib-cni-multus\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723427 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-socket-dir-parent\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.724546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723460 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/05ccb78a-325c-4951-b04d-d4b614127b78-serviceca\") pod \"node-ca-lrct7\" (UID: \"05ccb78a-325c-4951-b04d-d4b614127b78\") " pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-sysconfig\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723437 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-hostroot\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-socket-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4598f53c-2bba-424d-8c59-bec7a4697d98-hosts-file\") pod \"node-resolver-klpfw\" (UID: \"4598f53c-2bba-424d-8c59-bec7a4697d98\") " pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmhf\" (UniqueName: \"kubernetes.io/projected/05ccb78a-325c-4951-b04d-d4b614127b78-kube-api-access-rsmhf\") pod \"node-ca-lrct7\" (UID: \"05ccb78a-325c-4951-b04d-d4b614127b78\") " pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-var-lib-cni-bin\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-etc-kubernetes\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723688 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-etc-kubernetes\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723733 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-var-lib-cni-bin\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-var-lib-openvswitch\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-lib-modules\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftpnn\" (UniqueName: \"kubernetes.io/projected/f07b5bb3-bf4e-451f-9536-2824acccd8b3-kube-api-access-ftpnn\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4598f53c-2bba-424d-8c59-bec7a4697d98-tmp-dir\") pod \"node-resolver-klpfw\" (UID: \"4598f53c-2bba-424d-8c59-bec7a4697d98\") " pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mchjd\" (UniqueName: \"kubernetes.io/projected/210b36c9-4f24-42d9-a8aa-97ff81012c72-kube-api-access-mchjd\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05ccb78a-325c-4951-b04d-d4b614127b78-host\") pod \"node-ca-lrct7\" (UID: \"05ccb78a-325c-4951-b04d-d4b614127b78\") " pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-run-netns\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.727941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.723986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-run-openvswitch\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-log-socket\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724024 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-var-lib-openvswitch\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-systemd-units\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77c87b0b-9613-4070-a8e6-eea000f99c1a-ovnkube-config\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-systemd-units\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05ccb78a-325c-4951-b04d-d4b614127b78-host\") pod \"node-ca-lrct7\" (UID: \"05ccb78a-325c-4951-b04d-d4b614127b78\") " pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724218 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-run-netns\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724266 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-run-openvswitch\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-log-socket\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aee8d65f-586f-4879-a62f-69223df24c06-cni-binary-copy\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aee8d65f-586f-4879-a62f-69223df24c06-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-cnibin\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2519ea9e-2e8c-42a1-81c8-3649662d1c09-cni-binary-copy\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-run-netns\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.729132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-var-lib-kubelet\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724651 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77c87b0b-9613-4070-a8e6-eea000f99c1a-ovnkube-config\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724726 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-run-netns\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-run-ovn\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724781 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-var-lib-kubelet\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724793 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-run-ovn\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-cni-netd\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/514eec71-0eb2-4101-be06-0abbad141842-kubelet-config\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-os-release\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-systemd\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-run\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.724987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-var-lib-kubelet\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-host\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-tmp\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-registration-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-conf-dir\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-run-multus-certs\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.732121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-run-systemd\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77c87b0b-9613-4070-a8e6-eea000f99c1a-ovn-node-metrics-cert\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce9fa062-a9b3-4a71-a5c4-f76f22f36308-agent-certs\") pod \"konnectivity-agent-v8ln8\" (UID: \"ce9fa062-a9b3-4a71-a5c4-f76f22f36308\") " pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-device-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-sys-fs\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d803a3ed-0104-4caf-a879-2d5d851f31e6-host-slash\") pod \"iptables-alerter-dtbcc\" (UID: \"d803a3ed-0104-4caf-a879-2d5d851f31e6\") " pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2519ea9e-2e8c-42a1-81c8-3649662d1c09-cni-binary-copy\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-node-log\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-kubernetes\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvkjg\" (UniqueName: \"kubernetes.io/projected/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-kube-api-access-qvkjg\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-cni-dir\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-run-k8s-cni-cncf-io\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-cni-bin\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725733 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzqxt\" (UniqueName: \"kubernetes.io/projected/77c87b0b-9613-4070-a8e6-eea000f99c1a-kube-api-access-bzqxt\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.733051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725746 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhwq\" (UniqueName: \"kubernetes.io/projected/2519ea9e-2e8c-42a1-81c8-3649662d1c09-kube-api-access-4bhwq\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-modprobe-d\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725903 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-tuned\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aee8d65f-586f-4879-a62f-69223df24c06-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.725972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-slash\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmb8w\" (UniqueName: \"kubernetes.io/projected/4598f53c-2bba-424d-8c59-bec7a4697d98-kube-api-access-vmb8w\") pod \"node-resolver-klpfw\" (UID: \"4598f53c-2bba-424d-8c59-bec7a4697d98\") " pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zql72\" (UniqueName: \"kubernetes.io/projected/aee8d65f-586f-4879-a62f-69223df24c06-kube-api-access-zql72\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gg7\" (UniqueName: \"kubernetes.io/projected/d803a3ed-0104-4caf-a879-2d5d851f31e6-kube-api-access-p5gg7\") pod \"iptables-alerter-dtbcc\" (UID: \"d803a3ed-0104-4caf-a879-2d5d851f31e6\") " pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-system-cni-dir\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-kubelet\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-daemon-config\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-etc-openvswitch\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77c87b0b-9613-4070-a8e6-eea000f99c1a-env-overrides\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77c87b0b-9613-4070-a8e6-eea000f99c1a-ovnkube-script-lib\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.735495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-sysctl-d\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726392 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-run-k8s-cni-cncf-io\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726460 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-cni-bin\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726632 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-slash\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726693 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726693 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-conf-dir\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726726 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-host-run-multus-certs\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726766 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726778 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-system-cni-dir\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726807 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-run-systemd\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726837 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-kubelet\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726854 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-node-log\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.726986 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-cni-dir\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.727031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-host-cni-netd\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.727082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-cnibin\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.727354 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2519ea9e-2e8c-42a1-81c8-3649662d1c09-multus-daemon-config\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.727426 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77c87b0b-9613-4070-a8e6-eea000f99c1a-etc-openvswitch\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.727485 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2519ea9e-2e8c-42a1-81c8-3649662d1c09-os-release\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.736608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.727578 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77c87b0b-9613-4070-a8e6-eea000f99c1a-ovnkube-script-lib\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.738168 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.727850 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77c87b0b-9613-4070-a8e6-eea000f99c1a-env-overrides\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.738168 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.731617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77c87b0b-9613-4070-a8e6-eea000f99c1a-ovn-node-metrics-cert\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.739404 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.739174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzqxt\" (UniqueName: \"kubernetes.io/projected/77c87b0b-9613-4070-a8e6-eea000f99c1a-kube-api-access-bzqxt\") pod \"ovnkube-node-dsgzj\" (UID: \"77c87b0b-9613-4070-a8e6-eea000f99c1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.742457 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhwq\" (UniqueName: \"kubernetes.io/projected/2519ea9e-2e8c-42a1-81c8-3649662d1c09-kube-api-access-4bhwq\") pod \"multus-v9hsx\" (UID: \"2519ea9e-2e8c-42a1-81c8-3649662d1c09\") " pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.742987 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmhf\" (UniqueName: \"kubernetes.io/projected/05ccb78a-325c-4951-b04d-d4b614127b78-kube-api-access-rsmhf\") pod \"node-ca-lrct7\" (UID: \"05ccb78a-325c-4951-b04d-d4b614127b78\") " pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmb8w\" (UniqueName: \"kubernetes.io/projected/4598f53c-2bba-424d-8c59-bec7a4697d98-kube-api-access-vmb8w\") pod \"node-resolver-klpfw\" (UID: \"4598f53c-2bba-424d-8c59-bec7a4697d98\") " pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zql72\" (UniqueName: \"kubernetes.io/projected/aee8d65f-586f-4879-a62f-69223df24c06-kube-api-access-zql72\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gg7\" (UniqueName: \"kubernetes.io/projected/d803a3ed-0104-4caf-a879-2d5d851f31e6-kube-api-access-p5gg7\") pod \"iptables-alerter-dtbcc\" (UID: \"d803a3ed-0104-4caf-a879-2d5d851f31e6\") " pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-sysctl-d\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/514eec71-0eb2-4101-be06-0abbad141842-dbus\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-cnibin\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-sysctl-conf\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-sys\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-system-cni-dir\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-os-release\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d803a3ed-0104-4caf-a879-2d5d851f31e6-iptables-alerter-script\") pod \"iptables-alerter-dtbcc\" (UID: \"d803a3ed-0104-4caf-a879-2d5d851f31e6\") " pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce9fa062-a9b3-4a71-a5c4-f76f22f36308-konnectivity-ca\") pod \"konnectivity-agent-v8ln8\" (UID: \"ce9fa062-a9b3-4a71-a5c4-f76f22f36308\") " pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:08:56.833749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-sysconfig\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-socket-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4598f53c-2bba-424d-8c59-bec7a4697d98-hosts-file\") pod \"node-resolver-klpfw\" (UID: \"4598f53c-2bba-424d-8c59-bec7a4697d98\") " pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-lib-modules\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftpnn\" (UniqueName: \"kubernetes.io/projected/f07b5bb3-bf4e-451f-9536-2824acccd8b3-kube-api-access-ftpnn\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4598f53c-2bba-424d-8c59-bec7a4697d98-tmp-dir\") pod \"node-resolver-klpfw\" (UID: \"4598f53c-2bba-424d-8c59-bec7a4697d98\") " pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mchjd\" (UniqueName: \"kubernetes.io/projected/210b36c9-4f24-42d9-a8aa-97ff81012c72-kube-api-access-mchjd\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aee8d65f-586f-4879-a62f-69223df24c06-cni-binary-copy\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.827976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aee8d65f-586f-4879-a62f-69223df24c06-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/514eec71-0eb2-4101-be06-0abbad141842-kubelet-config\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-systemd\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-run\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-var-lib-kubelet\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-host\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-tmp\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.835149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-registration-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce9fa062-a9b3-4a71-a5c4-f76f22f36308-agent-certs\") pod \"konnectivity-agent-v8ln8\" (UID: \"ce9fa062-a9b3-4a71-a5c4-f76f22f36308\") " pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-device-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-sys-fs\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d803a3ed-0104-4caf-a879-2d5d851f31e6-host-slash\") pod \"iptables-alerter-dtbcc\" (UID: \"d803a3ed-0104-4caf-a879-2d5d851f31e6\") " pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-kubernetes\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvkjg\" (UniqueName: \"kubernetes.io/projected/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-kube-api-access-qvkjg\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-modprobe-d\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-tuned\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.828507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aee8d65f-586f-4879-a62f-69223df24c06-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aee8d65f-586f-4879-a62f-69223df24c06-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829363 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829432 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829587 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-cnibin\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829683 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-sysctl-conf\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.837742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829739 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-sysctl-d\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829761 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-sys\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829808 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/514eec71-0eb2-4101-be06-0abbad141842-dbus\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829809 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-system-cni-dir\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829856 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4598f53c-2bba-424d-8c59-bec7a4697d98-hosts-file\") pod \"node-resolver-klpfw\" (UID: \"4598f53c-2bba-424d-8c59-bec7a4697d98\") " pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.829900 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aee8d65f-586f-4879-a62f-69223df24c06-cni-binary-copy\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.829909 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.829984 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret podName:514eec71-0eb2-4101-be06-0abbad141842 nodeName:}" failed. No retries permitted until 2026-07-14 22:08:57.329962221 +0000 UTC m=+3.142593858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret") pod "global-pull-secret-syncer-f2j5k" (UID: "514eec71-0eb2-4101-be06-0abbad141842") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.830123 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.830174 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs podName:210b36c9-4f24-42d9-a8aa-97ff81012c72 nodeName:}" failed. No retries permitted until 2026-07-14 22:08:57.33015688 +0000 UTC m=+3.142788502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs") pod "network-metrics-daemon-wkv7z" (UID: "210b36c9-4f24-42d9-a8aa-97ff81012c72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d803a3ed-0104-4caf-a879-2d5d851f31e6-iptables-alerter-script\") pod \"iptables-alerter-dtbcc\" (UID: \"d803a3ed-0104-4caf-a879-2d5d851f31e6\") " pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830272 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-run\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830281 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-lib-modules\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830365 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-host\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830378 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-modprobe-d\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce9fa062-a9b3-4a71-a5c4-f76f22f36308-konnectivity-ca\") pod \"konnectivity-agent-v8ln8\" (UID: \"ce9fa062-a9b3-4a71-a5c4-f76f22f36308\") " pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:08:56.838843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830421 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4598f53c-2bba-424d-8c59-bec7a4697d98-tmp-dir\") pod \"node-resolver-klpfw\" (UID: \"4598f53c-2bba-424d-8c59-bec7a4697d98\") " pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830433 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-var-lib-kubelet\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830445 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-sysconfig\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-socket-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830488 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-systemd\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830493 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d803a3ed-0104-4caf-a879-2d5d851f31e6-host-slash\") pod \"iptables-alerter-dtbcc\" (UID: \"d803a3ed-0104-4caf-a879-2d5d851f31e6\") " pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830555 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-sys-fs\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830559 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-registration-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830609 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-kubernetes\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830664 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f07b5bb3-bf4e-451f-9536-2824acccd8b3-device-dir\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830718 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aee8d65f-586f-4879-a62f-69223df24c06-os-release\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830905 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aee8d65f-586f-4879-a62f-69223df24c06-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.830914 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/514eec71-0eb2-4101-be06-0abbad141842-kubelet-config\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.833300 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-etc-tuned\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.841847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.841300 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce9fa062-a9b3-4a71-a5c4-f76f22f36308-agent-certs\") pod \"konnectivity-agent-v8ln8\" (UID: \"ce9fa062-a9b3-4a71-a5c4-f76f22f36308\") " pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.845473 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" event={"ID":"04750a8bbdc612023d3422ff4bdfea8f","Type":"ContainerStarted","Data":"106e856f881b7d75cf9a310ab365c137b38eb7b3d1fc6a2598b71d938f2a88a9"} Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.846509 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.846529 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.846543 2578 projected.go:196] Error preparing data for projected volume kube-api-access-phh54 for pod openshift-network-diagnostics/network-check-target-fsbc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:56.846605 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54 podName:b0b82aef-9783-434b-9207-ae510b6dc0eb nodeName:}" failed. No retries permitted until 2026-07-14 22:08:57.346586814 +0000 UTC m=+3.159218461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-phh54" (UniqueName: "kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54") pod "network-check-target-fsbc5" (UID: "b0b82aef-9783-434b-9207-ae510b6dc0eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.847187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-tmp\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.849897 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftpnn\" (UniqueName: \"kubernetes.io/projected/f07b5bb3-bf4e-451f-9536-2824acccd8b3-kube-api-access-ftpnn\") pod \"aws-ebs-csi-driver-node-6hn74\" (UID: \"f07b5bb3-bf4e-451f-9536-2824acccd8b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.850249 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zql72\" (UniqueName: \"kubernetes.io/projected/aee8d65f-586f-4879-a62f-69223df24c06-kube-api-access-zql72\") pod \"multus-additional-cni-plugins-d6zkw\" (UID: \"aee8d65f-586f-4879-a62f-69223df24c06\") " pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.850886 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" event={"ID":"e71ccf9865e4302d818dc5edb3e82267","Type":"ContainerStarted","Data":"ba3f90932c8823378169858ab839b78d0ad37298ed3cf800ffef60dec5df602b"} Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.851753 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mchjd\" (UniqueName: \"kubernetes.io/projected/210b36c9-4f24-42d9-a8aa-97ff81012c72-kube-api-access-mchjd\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.852532 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvkjg\" (UniqueName: \"kubernetes.io/projected/ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d-kube-api-access-qvkjg\") pod \"tuned-lt2gz\" (UID: \"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d\") " pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.856096 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmb8w\" (UniqueName: \"kubernetes.io/projected/4598f53c-2bba-424d-8c59-bec7a4697d98-kube-api-access-vmb8w\") pod \"node-resolver-klpfw\" (UID: \"4598f53c-2bba-424d-8c59-bec7a4697d98\") " pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.884265 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gg7\" (UniqueName: \"kubernetes.io/projected/d803a3ed-0104-4caf-a879-2d5d851f31e6-kube-api-access-p5gg7\") pod \"iptables-alerter-dtbcc\" (UID: \"d803a3ed-0104-4caf-a879-2d5d851f31e6\") " pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.920015 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:08:56.929719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.928768 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v9hsx" Jul 14 22:08:56.935549 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:56.935515 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c87b0b_9613_4070_a8e6_eea000f99c1a.slice/crio-060b28c93185dc2d5862f0e712aad6bb609efed8423450d97729bc0e26153b96 WatchSource:0}: Error finding container 060b28c93185dc2d5862f0e712aad6bb609efed8423450d97729bc0e26153b96: Status 404 returned error can't find the container with id 060b28c93185dc2d5862f0e712aad6bb609efed8423450d97729bc0e26153b96 Jul 14 22:08:56.941196 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.940628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrct7" Jul 14 22:08:56.943524 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:56.943470 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2519ea9e_2e8c_42a1_81c8_3649662d1c09.slice/crio-f9c62a6444ab8f410c388028ecafe6bcb828938a74bd7edab3650e91b268d075 WatchSource:0}: Error finding container f9c62a6444ab8f410c388028ecafe6bcb828938a74bd7edab3650e91b268d075: Status 404 returned error can't find the container with id f9c62a6444ab8f410c388028ecafe6bcb828938a74bd7edab3650e91b268d075 Jul 14 22:08:56.948657 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.948471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" Jul 14 22:08:56.957353 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.957328 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dtbcc" Jul 14 22:08:56.966650 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.966488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:08:56.967523 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:56.967496 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee8d65f_586f_4879_a62f_69223df24c06.slice/crio-e3009b94aeb783852a64a5d76dbf29faa82efd60f733832d1a0cce9870d209eb WatchSource:0}: Error finding container e3009b94aeb783852a64a5d76dbf29faa82efd60f733832d1a0cce9870d209eb: Status 404 returned error can't find the container with id e3009b94aeb783852a64a5d76dbf29faa82efd60f733832d1a0cce9870d209eb Jul 14 22:08:56.978844 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:56.978827 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" Jul 14 22:08:56.983952 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:56.983925 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9fa062_a9b3_4a71_a5c4_f76f22f36308.slice/crio-a4d66cbd6ec34e4fd787d56d74d3ef7ed2f53fe110c4cb2fec6ae23a449eee71 WatchSource:0}: Error finding container a4d66cbd6ec34e4fd787d56d74d3ef7ed2f53fe110c4cb2fec6ae23a449eee71: Status 404 returned error can't find the container with id a4d66cbd6ec34e4fd787d56d74d3ef7ed2f53fe110c4cb2fec6ae23a449eee71 Jul 14 22:08:56.997270 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:56.997246 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba04ef6d_6c8f_41c4_9dc8_a649447cbd1d.slice/crio-5caea59e67bef457dfbc5dedd029eff0c66381379b748e6eca3fe696e4ed38a3 WatchSource:0}: Error finding container 5caea59e67bef457dfbc5dedd029eff0c66381379b748e6eca3fe696e4ed38a3: Status 404 returned error can't find the container with id 5caea59e67bef457dfbc5dedd029eff0c66381379b748e6eca3fe696e4ed38a3 Jul 14 22:08:57.002556 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.002509 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klpfw" Jul 14 22:08:57.003953 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.003931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" Jul 14 22:08:57.331491 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.331398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:57.331666 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.331501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:57.331666 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:57.331596 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:08:57.331666 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:57.331649 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:08:57.331833 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:57.331683 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret podName:514eec71-0eb2-4101-be06-0abbad141842 nodeName:}" failed. No retries permitted until 2026-07-14 22:08:58.331659305 +0000 UTC m=+4.144290932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret") pod "global-pull-secret-syncer-f2j5k" (UID: "514eec71-0eb2-4101-be06-0abbad141842") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:08:57.331833 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:57.331712 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs podName:210b36c9-4f24-42d9-a8aa-97ff81012c72 nodeName:}" failed. No retries permitted until 2026-07-14 22:08:58.331691454 +0000 UTC m=+4.144323104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs") pod "network-metrics-daemon-wkv7z" (UID: "210b36c9-4f24-42d9-a8aa-97ff81012c72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:08:57.432726 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.432696 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:08:57.432918 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:57.432891 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:08:57.432918 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:57.432922 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:08:57.433121 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:57.432936 2578 projected.go:196] Error preparing data for projected volume kube-api-access-phh54 for pod openshift-network-diagnostics/network-check-target-fsbc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:08:57.433121 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:57.433001 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54 podName:b0b82aef-9783-434b-9207-ae510b6dc0eb nodeName:}" failed. No retries permitted until 2026-07-14 22:08:58.432983249 +0000 UTC m=+4.245614887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-phh54" (UniqueName: "kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54") pod "network-check-target-fsbc5" (UID: "b0b82aef-9783-434b-9207-ae510b6dc0eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:08:57.662148 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.662015 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 22:03:56 +0000 UTC" deadline="2028-03-11 17:21:42.053936715 +0000 UTC" Jul 14 22:08:57.662148 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.662080 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14539h12m44.391860371s" Jul 14 22:08:57.831532 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.831501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:57.831724 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:57.831638 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:08:57.855536 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.855493 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrct7" event={"ID":"05ccb78a-325c-4951-b04d-d4b614127b78","Type":"ContainerStarted","Data":"3f689e49f58f51f5c0ebee37a38b3b61dd1b9a72216c7da5c1d04d4aa7527749"} Jul 14 22:08:57.857404 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.857343 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" event={"ID":"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d","Type":"ContainerStarted","Data":"5caea59e67bef457dfbc5dedd029eff0c66381379b748e6eca3fe696e4ed38a3"} Jul 14 22:08:57.859023 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.858995 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v8ln8" event={"ID":"ce9fa062-a9b3-4a71-a5c4-f76f22f36308","Type":"ContainerStarted","Data":"a4d66cbd6ec34e4fd787d56d74d3ef7ed2f53fe110c4cb2fec6ae23a449eee71"} Jul 14 22:08:57.860984 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.860956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dtbcc" event={"ID":"d803a3ed-0104-4caf-a879-2d5d851f31e6","Type":"ContainerStarted","Data":"cdf636b327f84e92f8a3426e297e66d9d41d37d3f31d7c39983a01314c9c3f6f"} Jul 14 22:08:57.862473 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.862420 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" event={"ID":"aee8d65f-586f-4879-a62f-69223df24c06","Type":"ContainerStarted","Data":"e3009b94aeb783852a64a5d76dbf29faa82efd60f733832d1a0cce9870d209eb"} Jul 14 22:08:57.864481 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.864456 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v9hsx" event={"ID":"2519ea9e-2e8c-42a1-81c8-3649662d1c09","Type":"ContainerStarted","Data":"f9c62a6444ab8f410c388028ecafe6bcb828938a74bd7edab3650e91b268d075"} Jul 14 22:08:57.867596 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:57.867500 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" event={"ID":"77c87b0b-9613-4070-a8e6-eea000f99c1a","Type":"ContainerStarted","Data":"060b28c93185dc2d5862f0e712aad6bb609efed8423450d97729bc0e26153b96"} Jul 14 22:08:58.340718 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:58.340679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:58.340944 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:58.340755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:58.340944 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.340894 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:08:58.341064 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.340955 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret podName:514eec71-0eb2-4101-be06-0abbad141842 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:00.3409365 +0000 UTC m=+6.153568139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret") pod "global-pull-secret-syncer-f2j5k" (UID: "514eec71-0eb2-4101-be06-0abbad141842") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:08:58.341395 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.341372 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:08:58.341480 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.341431 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs podName:210b36c9-4f24-42d9-a8aa-97ff81012c72 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:00.341413078 +0000 UTC m=+6.154044703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs") pod "network-metrics-daemon-wkv7z" (UID: "210b36c9-4f24-42d9-a8aa-97ff81012c72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:08:58.441917 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:58.441875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:08:58.442529 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.442475 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:08:58.442529 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.442508 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:08:58.442706 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.442537 2578 projected.go:196] Error preparing data for projected volume kube-api-access-phh54 for pod openshift-network-diagnostics/network-check-target-fsbc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:08:58.442706 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.442613 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54 podName:b0b82aef-9783-434b-9207-ae510b6dc0eb nodeName:}" failed. No retries permitted until 2026-07-14 22:09:00.442583059 +0000 UTC m=+6.255214701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-phh54" (UniqueName: "kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54") pod "network-check-target-fsbc5" (UID: "b0b82aef-9783-434b-9207-ae510b6dc0eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:08:58.523827 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:08:58.523375 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4598f53c_2bba_424d_8c59_bec7a4697d98.slice/crio-683bb5bf655fd4d7ca7781e77503269deb90096232823ef2b0903a3d3baceb31 WatchSource:0}: Error finding container 683bb5bf655fd4d7ca7781e77503269deb90096232823ef2b0903a3d3baceb31: Status 404 returned error can't find the container with id 683bb5bf655fd4d7ca7781e77503269deb90096232823ef2b0903a3d3baceb31 Jul 14 22:08:58.834247 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:58.833424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:08:58.834247 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.833554 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:08:58.834247 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:58.834050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:08:58.834247 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:58.834156 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:08:58.873640 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:58.873602 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" event={"ID":"f07b5bb3-bf4e-451f-9536-2824acccd8b3","Type":"ContainerStarted","Data":"08a5b0d87ea22d9e734c5561804f5766811b6a4d74d86461c0604679499d50f2"} Jul 14 22:08:58.876657 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:58.876418 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klpfw" event={"ID":"4598f53c-2bba-424d-8c59-bec7a4697d98","Type":"ContainerStarted","Data":"683bb5bf655fd4d7ca7781e77503269deb90096232823ef2b0903a3d3baceb31"} Jul 14 22:08:59.831038 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:08:59.831000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:08:59.831233 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:08:59.831157 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:00.360013 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:00.359970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:00.360581 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:00.360058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:00.360581 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.360193 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:00.360581 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.360263 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs podName:210b36c9-4f24-42d9-a8aa-97ff81012c72 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:04.360243172 +0000 UTC m=+10.172874800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs") pod "network-metrics-daemon-wkv7z" (UID: "210b36c9-4f24-42d9-a8aa-97ff81012c72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:00.360787 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.360681 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:00.360787 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.360730 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret podName:514eec71-0eb2-4101-be06-0abbad141842 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:04.36071501 +0000 UTC m=+10.173346639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret") pod "global-pull-secret-syncer-f2j5k" (UID: "514eec71-0eb2-4101-be06-0abbad141842") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:00.460684 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:00.460641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:00.460880 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.460852 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:00.460880 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.460878 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:00.461000 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.460892 2578 projected.go:196] Error preparing data for projected volume kube-api-access-phh54 for pod openshift-network-diagnostics/network-check-target-fsbc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:00.461000 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.460951 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54 podName:b0b82aef-9783-434b-9207-ae510b6dc0eb nodeName:}" failed. No retries permitted until 2026-07-14 22:09:04.460932625 +0000 UTC m=+10.273564261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-phh54" (UniqueName: "kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54") pod "network-check-target-fsbc5" (UID: "b0b82aef-9783-434b-9207-ae510b6dc0eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:00.831565 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:00.831486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:00.831745 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.831613 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:00.832456 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:00.831446 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:00.832456 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:00.832411 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:01.831455 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:01.830903 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:01.831455 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:01.831042 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:02.831896 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:02.831771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:02.831896 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:02.831771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:02.832626 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:02.832546 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:02.833060 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:02.832997 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:03.831868 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:03.831269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:03.831868 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:03.831505 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:04.397221 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:04.396993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:04.397221 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:04.397071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:04.397221 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.397143 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:04.397221 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.397183 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:04.397221 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.397209 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs podName:210b36c9-4f24-42d9-a8aa-97ff81012c72 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:12.397190193 +0000 UTC m=+18.209821819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs") pod "network-metrics-daemon-wkv7z" (UID: "210b36c9-4f24-42d9-a8aa-97ff81012c72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:04.397221 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.397227 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret podName:514eec71-0eb2-4101-be06-0abbad141842 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:12.397219683 +0000 UTC m=+18.209851301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret") pod "global-pull-secret-syncer-f2j5k" (UID: "514eec71-0eb2-4101-be06-0abbad141842") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:04.498152 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:04.498112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:04.498330 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.498263 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:04.498330 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.498282 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:04.498330 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.498295 2578 projected.go:196] Error preparing data for projected volume kube-api-access-phh54 for pod openshift-network-diagnostics/network-check-target-fsbc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:04.498478 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.498363 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54 podName:b0b82aef-9783-434b-9207-ae510b6dc0eb nodeName:}" failed. No retries permitted until 2026-07-14 22:09:12.498342571 +0000 UTC m=+18.310974211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-phh54" (UniqueName: "kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54") pod "network-check-target-fsbc5" (UID: "b0b82aef-9783-434b-9207-ae510b6dc0eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:04.832437 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:04.831805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:04.832437 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.831923 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:04.832437 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:04.831999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:04.832437 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:04.832086 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:05.831657 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:05.831406 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:05.831822 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:05.831771 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:06.831209 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:06.831172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:06.831741 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:06.831172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:06.831741 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:06.831319 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:06.831741 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:06.831383 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:07.831170 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:07.831133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:07.831342 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:07.831277 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:08.830776 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:08.830744 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:08.830943 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:08.830873 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:08.830943 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:08.830931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:08.831060 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:08.831039 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:09.831503 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:09.831462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:09.831929 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:09.831617 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:10.831508 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:10.831472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:10.831941 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:10.831611 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:10.831941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:10.831637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:10.831941 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:10.831722 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:11.831486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:11.831457 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:11.831693 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:11.831573 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:12.458553 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:12.458516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:12.458711 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:12.458583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:12.458711 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.458671 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:12.458829 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.458726 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:12.458829 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.458738 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs podName:210b36c9-4f24-42d9-a8aa-97ff81012c72 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:28.458717494 +0000 UTC m=+34.271349121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs") pod "network-metrics-daemon-wkv7z" (UID: "210b36c9-4f24-42d9-a8aa-97ff81012c72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:12.458829 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.458802 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret podName:514eec71-0eb2-4101-be06-0abbad141842 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:28.458782353 +0000 UTC m=+34.271413976 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret") pod "global-pull-secret-syncer-f2j5k" (UID: "514eec71-0eb2-4101-be06-0abbad141842") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:12.559829 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:12.559794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:12.560007 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.559938 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:12.560007 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.559955 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:12.560007 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.559964 2578 projected.go:196] Error preparing data for projected volume kube-api-access-phh54 for pod openshift-network-diagnostics/network-check-target-fsbc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:12.560141 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.560012 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54 podName:b0b82aef-9783-434b-9207-ae510b6dc0eb nodeName:}" failed. No retries permitted until 2026-07-14 22:09:28.559997497 +0000 UTC m=+34.372629114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-phh54" (UniqueName: "kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54") pod "network-check-target-fsbc5" (UID: "b0b82aef-9783-434b-9207-ae510b6dc0eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:12.831148 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:12.831115 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:12.831324 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.831235 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:12.831324 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:12.831269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:12.831452 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:12.831387 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:13.831410 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:13.831378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:13.831862 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:13.831489 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:14.831996 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:14.831964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:14.832475 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:14.832101 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:14.832475 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:14.832179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:14.832475 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:14.832289 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:15.831513 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:15.831483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:15.831683 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:15.831613 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:16.830988 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.830810 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:16.831619 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:16.831063 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:16.831619 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.830904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:16.831619 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:16.831149 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:16.911019 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.910988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" event={"ID":"e71ccf9865e4302d818dc5edb3e82267","Type":"ContainerStarted","Data":"d0b287704aad2003fd74bd51e92ea49e3bd5bb7c9021fa5f452934c77bd4f9fc"} Jul 14 22:09:16.912742 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.912717 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" event={"ID":"ba04ef6d-6c8f-41c4-9dc8-a649447cbd1d","Type":"ContainerStarted","Data":"2508ab775b5483b60209c87cb52d25b772a5762dbda860fa236dc55ddd8c3a16"} Jul 14 22:09:16.914079 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.914048 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v8ln8" event={"ID":"ce9fa062-a9b3-4a71-a5c4-f76f22f36308","Type":"ContainerStarted","Data":"ad9dda475f614f725ddbd78a7dfce01b3dad4e2d1931ed28a4706795f3a2a0a2"} Jul 14 22:09:16.915511 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.915493 2578 generic.go:356] "Generic (PLEG): container finished" podID="aee8d65f-586f-4879-a62f-69223df24c06" containerID="bc5259ff7f28875a0f4d010cd0bbf798823d342c51a1927a406094fd85d7c895" exitCode=0 Jul 14 22:09:16.915587 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.915560 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" event={"ID":"aee8d65f-586f-4879-a62f-69223df24c06","Type":"ContainerDied","Data":"bc5259ff7f28875a0f4d010cd0bbf798823d342c51a1927a406094fd85d7c895"} Jul 14 22:09:16.916929 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.916904 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v9hsx" event={"ID":"2519ea9e-2e8c-42a1-81c8-3649662d1c09","Type":"ContainerStarted","Data":"25a96815e69e27b66ef6ee3382be9b2b3d034d86ab29e0d598ca5a924b5f3ece"} Jul 14 22:09:16.918549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.918532 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" event={"ID":"77c87b0b-9613-4070-a8e6-eea000f99c1a","Type":"ContainerStarted","Data":"16b62e5bbeb833f2529627bfd849de85690477f970d193697515fcca3dfe8806"} Jul 14 22:09:16.918630 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.918594 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" event={"ID":"77c87b0b-9613-4070-a8e6-eea000f99c1a","Type":"ContainerStarted","Data":"3009c01403eee7bd2977d62ece8b1ba4e108605b35226a065de20f9be57c0f9a"} Jul 14 22:09:16.919955 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.919935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" event={"ID":"f07b5bb3-bf4e-451f-9536-2824acccd8b3","Type":"ContainerStarted","Data":"ea88ce4645ce44f1df6589e0bc66c3d502406e977cc8619a7b39359cedb9d7e4"} Jul 14 22:09:16.921272 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.921231 2578 generic.go:356] "Generic (PLEG): container finished" podID="04750a8bbdc612023d3422ff4bdfea8f" containerID="fce147b31d8b7d10a0b6d9d0f65d6be5018238a4bc659f82c8475a4ffc6abc36" exitCode=0 Jul 14 22:09:16.921272 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.921265 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" event={"ID":"04750a8bbdc612023d3422ff4bdfea8f","Type":"ContainerDied","Data":"fce147b31d8b7d10a0b6d9d0f65d6be5018238a4bc659f82c8475a4ffc6abc36"} Jul 14 22:09:16.922430 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.922413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrct7" event={"ID":"05ccb78a-325c-4951-b04d-d4b614127b78","Type":"ContainerStarted","Data":"f03ce3c4755cffa767778fa500b4170aa8509d470b7e2c62414475ff45eeb07a"} Jul 14 22:09:16.923652 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.923635 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klpfw" event={"ID":"4598f53c-2bba-424d-8c59-bec7a4697d98","Type":"ContainerStarted","Data":"b2c8432cad50179705abec21b19fb7848dafa4d91f4041dc2d9cebb0aeeb777b"} Jul 14 22:09:16.927063 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.927028 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal" podStartSLOduration=20.927003024 podStartE2EDuration="20.927003024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:16.926999969 +0000 UTC m=+22.739631611" watchObservedRunningTime="2026-07-14 22:09:16.927003024 +0000 UTC m=+22.739634646" Jul 14 22:09:16.959931 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.959894 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-klpfw" podStartSLOduration=4.319340896 podStartE2EDuration="21.959882271s" totalImagesPullingTime="17.640541375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:16.942714916 +0000 UTC m=+22.755346555" watchObservedRunningTime="2026-07-14 22:09:16.959882271 +0000 UTC m=+22.772513975" Jul 14 22:09:16.960054 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.960016 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v9hsx" podStartSLOduration=2.736802391 podStartE2EDuration="21.960008616s" totalImagesPullingTime="19.223206225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:16.959693204 +0000 UTC m=+22.772324845" watchObservedRunningTime="2026-07-14 22:09:16.960008616 +0000 UTC m=+22.772640258" Jul 14 22:09:16.975185 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.975153 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v8ln8" podStartSLOduration=2.88481068 podStartE2EDuration="21.975140118s" totalImagesPullingTime="19.090329438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:16.974573887 +0000 UTC m=+22.787205527" watchObservedRunningTime="2026-07-14 22:09:16.975140118 +0000 UTC m=+22.787771758" Jul 14 22:09:16.990501 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:16.990463 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lrct7" podStartSLOduration=12.452648014 podStartE2EDuration="21.990452211s" totalImagesPullingTime="9.537804197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:16.989950192 +0000 UTC m=+22.802581833" watchObservedRunningTime="2026-07-14 22:09:16.990452211 +0000 UTC m=+22.803083850" Jul 14 22:09:17.062905 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:17.062739 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lt2gz" podStartSLOduration=2.926726418 podStartE2EDuration="22.062724143s" totalImagesPullingTime="19.135997725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:17.062329374 +0000 UTC m=+22.874961015" watchObservedRunningTime="2026-07-14 22:09:17.062724143 +0000 UTC m=+22.875355810" Jul 14 22:09:17.831087 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:17.831048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:17.831548 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:17.831173 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:17.929941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:17.929900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" event={"ID":"77c87b0b-9613-4070-a8e6-eea000f99c1a","Type":"ContainerStarted","Data":"05821b5b7e53e3ca2945ae79b67b2fe8e87a09803d7afe64bf750a8768d4beec"} Jul 14 22:09:17.930117 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:17.929991 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" event={"ID":"77c87b0b-9613-4070-a8e6-eea000f99c1a","Type":"ContainerStarted","Data":"cd1a23b6330d0c626aa75d45bab0f080ac61d2f3b6447ed1bcebd13f1bb92242"} Jul 14 22:09:17.930117 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:17.930041 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" event={"ID":"77c87b0b-9613-4070-a8e6-eea000f99c1a","Type":"ContainerStarted","Data":"334fc686d7f78ae52498992edeebaf776d771b64051f203921815b990a519027"} Jul 14 22:09:17.930117 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:17.930088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" event={"ID":"77c87b0b-9613-4070-a8e6-eea000f99c1a","Type":"ContainerStarted","Data":"e7b4602120c3b419628313c9ac594e6a80ea8bb034f0236f27d1d8e8ddbfe780"} Jul 14 22:09:17.932470 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:17.932313 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dtbcc" event={"ID":"d803a3ed-0104-4caf-a879-2d5d851f31e6","Type":"ContainerStarted","Data":"b7f8950e1aca6766d5128998c6aafc2bc7c78dd927ee57ff2cd53dd3c2ccf214"} Jul 14 22:09:17.948547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:17.948506 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dtbcc" podStartSLOduration=4.019361995 podStartE2EDuration="22.948489061s" totalImagesPullingTime="18.929127066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:17.948021691 +0000 UTC m=+23.760653332" watchObservedRunningTime="2026-07-14 22:09:17.948489061 +0000 UTC m=+23.761120701" Jul 14 22:09:18.112262 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:18.112204 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 22:09:18.700450 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:18.700258 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T22:09:18.112224009Z","UUID":"c3c67d78-b0c4-4e1d-aea2-fe37ae96ea6c","Handler":null,"Name":"","Endpoint":""} Jul 14 22:09:18.701752 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:18.701724 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 22:09:18.701887 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:18.701772 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 22:09:18.831711 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:18.831683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:18.832159 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:18.831682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:18.832159 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:18.831819 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:18.832159 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:18.831863 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:18.935389 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:18.935350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" event={"ID":"f07b5bb3-bf4e-451f-9536-2824acccd8b3","Type":"ContainerStarted","Data":"fb84be7cf9e9e3f89c4a7e14918ea2fd59b355e3fb843b83f41af2406e4564c7"} Jul 14 22:09:18.937295 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:18.937267 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" event={"ID":"04750a8bbdc612023d3422ff4bdfea8f","Type":"ContainerStarted","Data":"efd07d0fc288940ecbe68b7d093aa7e798e4987d348fe2137d0c398b1aa47c38"} Jul 14 22:09:18.953733 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:18.953654 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-45.ec2.internal" podStartSLOduration=22.953637967 podStartE2EDuration="22.953637967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:18.953516018 +0000 UTC m=+24.766147658" watchObservedRunningTime="2026-07-14 22:09:18.953637967 +0000 UTC m=+24.766269608" Jul 14 22:09:19.831783 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:19.831755 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:19.832193 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:19.831873 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:19.944040 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:19.944006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" event={"ID":"77c87b0b-9613-4070-a8e6-eea000f99c1a","Type":"ContainerStarted","Data":"4dc200163bdcead0f074e5bde6281444dcbef52f99907e57f6c2205551b6d609"} Jul 14 22:09:19.946884 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:19.946843 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" event={"ID":"f07b5bb3-bf4e-451f-9536-2824acccd8b3","Type":"ContainerStarted","Data":"35489c4327919e20a223a93c05d4007b47a2aa94ce1f076bc0b3c1bf3950243a"} Jul 14 22:09:19.969566 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:19.969516 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6hn74" podStartSLOduration=5.076218791 podStartE2EDuration="24.969498826s" totalImagesPullingTime="19.893280035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:19.96933115 +0000 UTC m=+25.781962793" watchObservedRunningTime="2026-07-14 22:09:19.969498826 +0000 UTC m=+25.782130468" Jul 14 22:09:20.831383 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:20.831350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:20.831566 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:20.831350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:20.831566 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:20.831471 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:20.831566 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:20.831546 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:21.618419 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.618102 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:09:21.619031 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.618695 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:09:21.830928 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.830855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:21.831066 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:21.830971 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:21.952886 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.952850 2578 generic.go:356] "Generic (PLEG): container finished" podID="aee8d65f-586f-4879-a62f-69223df24c06" containerID="50688b40f2686ad58d364fda8b18a4988fc6d8accb1edfd56636bb580aac0e91" exitCode=0 Jul 14 22:09:21.953058 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.952922 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" event={"ID":"aee8d65f-586f-4879-a62f-69223df24c06","Type":"ContainerDied","Data":"50688b40f2686ad58d364fda8b18a4988fc6d8accb1edfd56636bb580aac0e91"} Jul 14 22:09:21.956776 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.956754 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" event={"ID":"77c87b0b-9613-4070-a8e6-eea000f99c1a","Type":"ContainerStarted","Data":"4eb59aa72f4091375e3d82afde125a74a3ef6fdfd24bfe99ed4fdcebaedd5dac"} Jul 14 22:09:21.957057 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.957037 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:09:21.957170 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.957114 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:09:21.957170 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.957145 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:09:21.972095 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.972073 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:09:21.987223 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:21.987204 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:09:22.003547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:22.003509 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" podStartSLOduration=7.303201059 podStartE2EDuration="27.003494596s" totalImagesPullingTime="19.700293537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:22.001418405 +0000 UTC m=+27.814050044" watchObservedRunningTime="2026-07-14 22:09:22.003494596 +0000 UTC m=+27.816126236" Jul 14 22:09:22.831484 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:22.831221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:22.832363 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:22.831296 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:22.832363 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:22.831602 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:22.832363 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:22.832042 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:23.538991 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:23.538916 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2j5k"] Jul 14 22:09:23.539126 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:23.539019 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:23.539126 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:23.539107 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:23.544903 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:23.544876 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fsbc5"] Jul 14 22:09:23.545022 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:23.544995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:23.545109 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:23.545090 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:23.545518 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:23.545499 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wkv7z"] Jul 14 22:09:23.545591 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:23.545581 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:23.545690 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:23.545667 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:23.962211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:23.962176 2578 generic.go:356] "Generic (PLEG): container finished" podID="aee8d65f-586f-4879-a62f-69223df24c06" containerID="4620b877a0e7b6522ad2603077f7ee8499b6a68b17ea1d438719d69f3c9a407d" exitCode=0 Jul 14 22:09:23.962562 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:23.962273 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" event={"ID":"aee8d65f-586f-4879-a62f-69223df24c06","Type":"ContainerDied","Data":"4620b877a0e7b6522ad2603077f7ee8499b6a68b17ea1d438719d69f3c9a407d"} Jul 14 22:09:24.832455 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:24.832417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:24.832629 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:24.832509 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:24.832687 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:24.832620 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:24.832687 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:24.832511 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:25.830853 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:25.830777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:25.831184 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:25.830872 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:25.969396 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:25.969360 2578 generic.go:356] "Generic (PLEG): container finished" podID="aee8d65f-586f-4879-a62f-69223df24c06" containerID="baeaae47c3d93447f834d773da226573e43b147f2aeee66ec5350047dad953f7" exitCode=0 Jul 14 22:09:25.969536 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:25.969422 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" event={"ID":"aee8d65f-586f-4879-a62f-69223df24c06","Type":"ContainerDied","Data":"baeaae47c3d93447f834d773da226573e43b147f2aeee66ec5350047dad953f7"} Jul 14 22:09:26.811500 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:26.811464 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:09:26.811670 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:26.811654 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 22:09:26.812125 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:26.812102 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v8ln8" Jul 14 22:09:26.830955 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:26.830930 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:26.831323 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:26.831054 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:26.831323 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:26.831112 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:26.831323 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:26.831236 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:27.831161 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:27.831132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:27.831605 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:27.831261 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:28.489341 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:28.489104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:28.489341 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.489281 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:28.489341 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:28.489335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:28.489574 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.489355 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs podName:210b36c9-4f24-42d9-a8aa-97ff81012c72 nodeName:}" failed. No retries permitted until 2026-07-14 22:10:00.489332005 +0000 UTC m=+66.301963623 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs") pod "network-metrics-daemon-wkv7z" (UID: "210b36c9-4f24-42d9-a8aa-97ff81012c72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:28.489574 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.489412 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:28.489574 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.489459 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret podName:514eec71-0eb2-4101-be06-0abbad141842 nodeName:}" failed. No retries permitted until 2026-07-14 22:10:00.489447111 +0000 UTC m=+66.302078729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret") pod "global-pull-secret-syncer-f2j5k" (UID: "514eec71-0eb2-4101-be06-0abbad141842") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:28.590100 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:28.590062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:28.590277 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.590219 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:28.590277 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.590262 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:28.590277 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.590278 2578 projected.go:196] Error preparing data for projected volume kube-api-access-phh54 for pod openshift-network-diagnostics/network-check-target-fsbc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:28.590404 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.590345 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54 podName:b0b82aef-9783-434b-9207-ae510b6dc0eb nodeName:}" failed. No retries permitted until 2026-07-14 22:10:00.59032311 +0000 UTC m=+66.402954751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-phh54" (UniqueName: "kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54") pod "network-check-target-fsbc5" (UID: "b0b82aef-9783-434b-9207-ae510b6dc0eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:28.830948 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:28.830912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:28.831106 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:28.830912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:28.831106 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.831046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2j5k" podUID="514eec71-0eb2-4101-be06-0abbad141842" Jul 14 22:09:28.831106 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:28.831100 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkv7z" podUID="210b36c9-4f24-42d9-a8aa-97ff81012c72" Jul 14 22:09:29.831350 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:29.831331 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:29.831604 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:29.831423 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fsbc5" podUID="b0b82aef-9783-434b-9207-ae510b6dc0eb" Jul 14 22:09:29.982278 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:29.982222 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" event={"ID":"aee8d65f-586f-4879-a62f-69223df24c06","Type":"ContainerStarted","Data":"4965d149c8d3396b715ad022cb7540a105e00de315eb6bfc4f8cc8e31b4e1eea"} Jul 14 22:09:30.454090 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.454062 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-45.ec2.internal" event="NodeReady" Jul 14 22:09:30.454284 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.454192 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 22:09:30.505896 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.505865 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-6z66z"] Jul 14 22:09:30.508812 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.508792 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q"] Jul 14 22:09:30.508926 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.508904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.512288 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.512269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.519067 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.519043 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.519191 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.519049 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 14 22:09:30.522956 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.522939 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.524014 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.523996 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 14 22:09:30.524158 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.524016 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.525024 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.525010 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.525114 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.525034 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 14 22:09:30.525274 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.525260 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-m9snc\"" Jul 14 22:09:30.525687 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.525672 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 14 22:09:30.526139 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.526123 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gnf4s\"" Jul 14 22:09:30.529910 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.529890 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv"] Jul 14 22:09:30.533552 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.533525 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz"] Jul 14 22:09:30.533743 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.533728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.535860 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.535843 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 14 22:09:30.536926 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.536912 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt"] Jul 14 22:09:30.537027 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.537014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:30.540050 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.540029 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6b494b4dd8-57vzj"] Jul 14 22:09:30.540137 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.540109 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt" Jul 14 22:09:30.543198 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.543182 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-fmwwd"] Jul 14 22:09:30.543348 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.543332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.546255 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.546226 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995"] Jul 14 22:09:30.546337 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.546320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.549196 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.549183 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2"] Jul 14 22:09:30.549314 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.549301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:30.552161 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.552147 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq"] Jul 14 22:09:30.552295 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.552280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2" Jul 14 22:09:30.555096 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.555080 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-6z66z"] Jul 14 22:09:30.555195 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.555163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.556938 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.556920 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 22:09:30.556938 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.556932 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 22:09:30.557880 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.557780 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 22:09:30.568618 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.568595 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 14 22:09:30.571060 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571037 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gv7p9\"" Jul 14 22:09:30.571060 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571046 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.571210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571119 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vznsg\"" Jul 14 22:09:30.571462 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571441 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.571555 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571495 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 14 22:09:30.571555 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571550 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.571674 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571453 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 14 22:09:30.571804 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571787 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.571869 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571791 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 14 22:09:30.571869 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.571859 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.572550 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.572534 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 14 22:09:30.572716 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.572699 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 14 22:09:30.572838 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.572822 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 14 22:09:30.572915 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.572864 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.572972 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.572938 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.573012 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.572958 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.573098 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.573011 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.573157 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.573128 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.574451 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.574431 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-29dxq\"" Jul 14 22:09:30.574547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.574486 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k2gvh\"" Jul 14 22:09:30.574612 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.574561 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 14 22:09:30.574667 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.574651 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.574713 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.574689 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.574804 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.574782 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fltwm\"" Jul 14 22:09:30.574900 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.574885 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 14 22:09:30.574967 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.574904 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 14 22:09:30.575024 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.574997 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hff8g\"" Jul 14 22:09:30.575138 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.575123 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pl2nd\"" Jul 14 22:09:30.575328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.575312 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.575430 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.575327 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 14 22:09:30.575430 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.575408 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9dssp\"" Jul 14 22:09:30.575542 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.575491 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.581774 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.581753 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q"] Jul 14 22:09:30.582135 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.582113 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 14 22:09:30.582667 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.582650 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-fmwwd"] Jul 14 22:09:30.583388 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.583367 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 22:09:30.584531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.584516 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2"] Jul 14 22:09:30.585464 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.585450 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq"] Jul 14 22:09:30.586424 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.586411 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz"] Jul 14 22:09:30.587273 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.587252 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv"] Jul 14 22:09:30.593844 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.593823 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b494b4dd8-57vzj"] Jul 14 22:09:30.600063 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.600043 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt"] Jul 14 22:09:30.612198 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.612179 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995"] Jul 14 22:09:30.666870 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.666847 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-55p9j"] Jul 14 22:09:30.670011 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.669987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.673665 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.673650 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 22:09:30.673911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.673899 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-29nlw\"" Jul 14 22:09:30.673958 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.673940 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 22:09:30.689290 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.689271 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-55p9j"] Jul 14 22:09:30.692823 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.692801 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5mbjm"] Jul 14 22:09:30.698213 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.698195 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5mbjm" Jul 14 22:09:30.704349 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.704309 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 22:09:30.704579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.704564 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 22:09:30.704648 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.704624 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 22:09:30.704734 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.704721 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jl5db\"" Jul 14 22:09:30.707803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.707786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w8bg\" (UniqueName: \"kubernetes.io/projected/92e2a1a7-c30d-4973-a60a-a21ac171201a-kube-api-access-6w8bg\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:30.707856 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.707825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc87cee2-2ef1-4d66-977f-47f83f7cc504-config\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.707856 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.707845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:30.707920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.707886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ca2dc15-2a22-4ed0-b950-db1983dc6fc4-serving-cert\") pod \"service-ca-operator-7695fc6c8c-q2g8q\" (UID: \"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.707920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.707909 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nv9q\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-kube-api-access-7nv9q\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.708004 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.707929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-trusted-ca\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.708004 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.707960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-stats-auth\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.708070 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95fe832f-c448-4835-bdd6-2c03d94edc5f-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-66wfq\" (UID: \"95fe832f-c448-4835-bdd6-2c03d94edc5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.708070 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-default-certificate\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.708070 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phngg\" (UniqueName: \"kubernetes.io/projected/f4aa4416-888a-403e-9a78-07bd8ec23af4-kube-api-access-phngg\") pod \"network-check-source-869dcfcdcd-8vrr2\" (UID: \"f4aa4416-888a-403e-9a78-07bd8ec23af4\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2" Jul 14 22:09:30.708205 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/92e2a1a7-c30d-4973-a60a-a21ac171201a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:30.708205 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-bound-sa-token\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.708205 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnpc\" (UniqueName: \"kubernetes.io/projected/c78ebcf5-0805-48e9-b946-4c759026f110-kube-api-access-flnpc\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.708205 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc87cee2-2ef1-4d66-977f-47f83f7cc504-trusted-ca\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.708205 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-installation-pull-secrets\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-image-registry-private-configuration\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78ebcf5-0805-48e9-b946-4c759026f110-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnpm8\" (UniqueName: \"kubernetes.io/projected/b952986a-e1d3-4ea4-86f0-6411a8a41eed-kube-api-access-bnpm8\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc87cee2-2ef1-4d66-977f-47f83f7cc504-serving-cert\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jbf7\" (UniqueName: \"kubernetes.io/projected/dc87cee2-2ef1-4d66-977f-47f83f7cc504-kube-api-access-6jbf7\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddhpv\" (UniqueName: \"kubernetes.io/projected/b093ad9b-7ff0-458a-8208-6356b7a1332b-kube-api-access-ddhpv\") pod \"volume-data-source-validator-78495597f6-ts2bt\" (UID: \"b093ad9b-7ff0-458a-8208-6356b7a1332b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455ccbb-3529-44c1-9c6d-ac80be15648f-ca-trust-extracted\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.708461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zksmz\" (UniqueName: \"kubernetes.io/projected/95fe832f-c448-4835-bdd6-2c03d94edc5f-kube-api-access-zksmz\") pod \"kube-storage-version-migrator-operator-799cb7f958-66wfq\" (UID: \"95fe832f-c448-4835-bdd6-2c03d94edc5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-certificates\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c78ebcf5-0805-48e9-b946-4c759026f110-tmp\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95fe832f-c448-4835-bdd6-2c03d94edc5f-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-66wfq\" (UID: \"95fe832f-c448-4835-bdd6-2c03d94edc5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ca2dc15-2a22-4ed0-b950-db1983dc6fc4-config\") pod \"service-ca-operator-7695fc6c8c-q2g8q\" (UID: \"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78ebcf5-0805-48e9-b946-4c759026f110-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvcb\" (UniqueName: \"kubernetes.io/projected/6ca2dc15-2a22-4ed0-b950-db1983dc6fc4-kube-api-access-jbvcb\") pod \"service-ca-operator-7695fc6c8c-q2g8q\" (UID: \"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drhl\" (UniqueName: \"kubernetes.io/projected/786291b9-f5ed-48f9-84af-097f93692bfe-kube-api-access-2drhl\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708665 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c78ebcf5-0805-48e9-b946-4c759026f110-snapshots\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.708920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.708682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c78ebcf5-0805-48e9-b946-4c759026f110-serving-cert\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.709656 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.709640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5mbjm"] Jul 14 22:09:30.736408 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.736381 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:30.809149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.809286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78ebcf5-0805-48e9-b946-4c759026f110-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.809286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvcb\" (UniqueName: \"kubernetes.io/projected/6ca2dc15-2a22-4ed0-b950-db1983dc6fc4-kube-api-access-jbvcb\") pod \"service-ca-operator-7695fc6c8c-q2g8q\" (UID: \"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.809286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.809286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drhl\" (UniqueName: \"kubernetes.io/projected/786291b9-f5ed-48f9-84af-097f93692bfe-kube-api-access-2drhl\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:30.809286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c78ebcf5-0805-48e9-b946-4c759026f110-snapshots\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.809286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c78ebcf5-0805-48e9-b946-4c759026f110-serving-cert\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.809525 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0784199-9bd0-4ecf-a5d5-ef31376c45a8-cert\") pod \"ingress-canary-5mbjm\" (UID: \"d0784199-9bd0-4ecf-a5d5-ef31376c45a8\") " pod="openshift-ingress-canary/ingress-canary-5mbjm" Jul 14 22:09:30.809525 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.809337 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:31.309315821 +0000 UTC m=+37.121947457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : configmap references non-existent config key: service-ca.crt Jul 14 22:09:30.809525 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.809348 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 22:09:30.809525 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.809366 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv: secret "image-registry-tls" not found Jul 14 22:09:30.809525 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.809438 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls podName:6455ccbb-3529-44c1-9c6d-ac80be15648f nodeName:}" failed. No retries permitted until 2026-07-14 22:09:31.309414985 +0000 UTC m=+37.122046605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls") pod "image-registry-7c5bd5fbf5-xh6bv" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f") : secret "image-registry-tls" not found Jul 14 22:09:30.809525 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8bg\" (UniqueName: \"kubernetes.io/projected/92e2a1a7-c30d-4973-a60a-a21ac171201a-kube-api-access-6w8bg\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:30.809525 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc87cee2-2ef1-4d66-977f-47f83f7cc504-config\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ca2dc15-2a22-4ed0-b950-db1983dc6fc4-serving-cert\") pod \"service-ca-operator-7695fc6c8c-q2g8q\" (UID: \"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nv9q\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-kube-api-access-7nv9q\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-trusted-ca\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.809627 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.809672 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls podName:92e2a1a7-c30d-4973-a60a-a21ac171201a nodeName:}" failed. No retries permitted until 2026-07-14 22:09:31.309658427 +0000 UTC m=+37.122290056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-w2jcz" (UID: "92e2a1a7-c30d-4973-a60a-a21ac171201a") : secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-stats-auth\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc2b4f2-7551-4c24-95c1-21388f68f67b-tmp-dir\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95fe832f-c448-4835-bdd6-2c03d94edc5f-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-66wfq\" (UID: \"95fe832f-c448-4835-bdd6-2c03d94edc5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-default-certificate\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phngg\" (UniqueName: \"kubernetes.io/projected/f4aa4416-888a-403e-9a78-07bd8ec23af4-kube-api-access-phngg\") pod \"network-check-source-869dcfcdcd-8vrr2\" (UID: \"f4aa4416-888a-403e-9a78-07bd8ec23af4\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2" Jul 14 22:09:30.809827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/92e2a1a7-c30d-4973-a60a-a21ac171201a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:30.810396 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-bound-sa-token\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.810396 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnpc\" (UniqueName: \"kubernetes.io/projected/c78ebcf5-0805-48e9-b946-4c759026f110-kube-api-access-flnpc\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.810396 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.809894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc87cee2-2ef1-4d66-977f-47f83f7cc504-trusted-ca\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.810396 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.810314 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc87cee2-2ef1-4d66-977f-47f83f7cc504-config\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.810583 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.810421 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95fe832f-c448-4835-bdd6-2c03d94edc5f-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-66wfq\" (UID: \"95fe832f-c448-4835-bdd6-2c03d94edc5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.810795 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.810769 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-trusted-ca\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.810884 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.810805 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc87cee2-2ef1-4d66-977f-47f83f7cc504-trusted-ca\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.810938 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.810913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:30.810974 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.810946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-installation-pull-secrets\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.811024 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.810972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qktr4\" (UniqueName: \"kubernetes.io/projected/9dc2b4f2-7551-4c24-95c1-21388f68f67b-kube-api-access-qktr4\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.811024 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dzg\" (UniqueName: \"kubernetes.io/projected/d0784199-9bd0-4ecf-a5d5-ef31376c45a8-kube-api-access-b6dzg\") pod \"ingress-canary-5mbjm\" (UID: \"d0784199-9bd0-4ecf-a5d5-ef31376c45a8\") " pod="openshift-ingress-canary/ingress-canary-5mbjm" Jul 14 22:09:30.811171 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-image-registry-private-configuration\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.811171 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78ebcf5-0805-48e9-b946-4c759026f110-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.811171 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.811080 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 22:09:30.811171 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnpm8\" (UniqueName: \"kubernetes.io/projected/b952986a-e1d3-4ea4-86f0-6411a8a41eed-kube-api-access-bnpm8\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.811171 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc87cee2-2ef1-4d66-977f-47f83f7cc504-serving-cert\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.811171 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.811132 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls podName:786291b9-f5ed-48f9-84af-097f93692bfe nodeName:}" failed. No retries permitted until 2026-07-14 22:09:31.311115164 +0000 UTC m=+37.123746798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls") pod "cluster-samples-operator-65db858894-f9995" (UID: "786291b9-f5ed-48f9-84af-097f93692bfe") : secret "samples-operator-tls" not found Jul 14 22:09:30.811486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jbf7\" (UniqueName: \"kubernetes.io/projected/dc87cee2-2ef1-4d66-977f-47f83f7cc504-kube-api-access-6jbf7\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.811486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811230 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/92e2a1a7-c30d-4973-a60a-a21ac171201a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:30.811486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhpv\" (UniqueName: \"kubernetes.io/projected/b093ad9b-7ff0-458a-8208-6356b7a1332b-kube-api-access-ddhpv\") pod \"volume-data-source-validator-78495597f6-ts2bt\" (UID: \"b093ad9b-7ff0-458a-8208-6356b7a1332b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt" Jul 14 22:09:30.811486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.811486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455ccbb-3529-44c1-9c6d-ac80be15648f-ca-trust-extracted\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.811486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksmz\" (UniqueName: \"kubernetes.io/projected/95fe832f-c448-4835-bdd6-2c03d94edc5f-kube-api-access-zksmz\") pod \"kube-storage-version-migrator-operator-799cb7f958-66wfq\" (UID: \"95fe832f-c448-4835-bdd6-2c03d94edc5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.811486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-certificates\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.811486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.811486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811472 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c78ebcf5-0805-48e9-b946-4c759026f110-snapshots\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.811897 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c78ebcf5-0805-48e9-b946-4c759026f110-tmp\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.811969 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95fe832f-c448-4835-bdd6-2c03d94edc5f-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-66wfq\" (UID: \"95fe832f-c448-4835-bdd6-2c03d94edc5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.812020 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.811992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ca2dc15-2a22-4ed0-b950-db1983dc6fc4-config\") pod \"service-ca-operator-7695fc6c8c-q2g8q\" (UID: \"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.812070 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.812055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc2b4f2-7551-4c24-95c1-21388f68f67b-config-volume\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.812138 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.812121 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 22:09:30.812187 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.812140 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c78ebcf5-0805-48e9-b946-4c759026f110-tmp\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.812187 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.812141 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-certificates\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.812294 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.812221 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:31.312205167 +0000 UTC m=+37.124836788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : secret "router-metrics-certs-default" not found Jul 14 22:09:30.812724 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.812697 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ca2dc15-2a22-4ed0-b950-db1983dc6fc4-config\") pod \"service-ca-operator-7695fc6c8c-q2g8q\" (UID: \"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.812840 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.812817 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455ccbb-3529-44c1-9c6d-ac80be15648f-ca-trust-extracted\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.812901 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.812839 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78ebcf5-0805-48e9-b946-4c759026f110-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.813487 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.813464 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78ebcf5-0805-48e9-b946-4c759026f110-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.814704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.814678 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc87cee2-2ef1-4d66-977f-47f83f7cc504-serving-cert\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.814788 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.814756 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-stats-auth\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.814891 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.814863 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ca2dc15-2a22-4ed0-b950-db1983dc6fc4-serving-cert\") pod \"service-ca-operator-7695fc6c8c-q2g8q\" (UID: \"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.814993 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.814908 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-image-registry-private-configuration\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.814993 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.814938 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-default-certificate\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.815080 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.815007 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95fe832f-c448-4835-bdd6-2c03d94edc5f-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-66wfq\" (UID: \"95fe832f-c448-4835-bdd6-2c03d94edc5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.815270 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.815233 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-installation-pull-secrets\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.815631 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.815616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c78ebcf5-0805-48e9-b946-4c759026f110-serving-cert\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.825658 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.825628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drhl\" (UniqueName: \"kubernetes.io/projected/786291b9-f5ed-48f9-84af-097f93692bfe-kube-api-access-2drhl\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:30.825852 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.825837 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nv9q\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-kube-api-access-7nv9q\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.826732 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.826714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8bg\" (UniqueName: \"kubernetes.io/projected/92e2a1a7-c30d-4973-a60a-a21ac171201a-kube-api-access-6w8bg\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:30.829998 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.829970 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhpv\" (UniqueName: \"kubernetes.io/projected/b093ad9b-7ff0-458a-8208-6356b7a1332b-kube-api-access-ddhpv\") pod \"volume-data-source-validator-78495597f6-ts2bt\" (UID: \"b093ad9b-7ff0-458a-8208-6356b7a1332b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt" Jul 14 22:09:30.830096 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.830072 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phngg\" (UniqueName: \"kubernetes.io/projected/f4aa4416-888a-403e-9a78-07bd8ec23af4-kube-api-access-phngg\") pod \"network-check-source-869dcfcdcd-8vrr2\" (UID: \"f4aa4416-888a-403e-9a78-07bd8ec23af4\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2" Jul 14 22:09:30.830162 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.830133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnpm8\" (UniqueName: \"kubernetes.io/projected/b952986a-e1d3-4ea4-86f0-6411a8a41eed-kube-api-access-bnpm8\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:30.830939 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.830800 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksmz\" (UniqueName: \"kubernetes.io/projected/95fe832f-c448-4835-bdd6-2c03d94edc5f-kube-api-access-zksmz\") pod \"kube-storage-version-migrator-operator-799cb7f958-66wfq\" (UID: \"95fe832f-c448-4835-bdd6-2c03d94edc5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.830939 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.830881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:09:30.830939 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.830904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:09:30.831141 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.831038 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvcb\" (UniqueName: \"kubernetes.io/projected/6ca2dc15-2a22-4ed0-b950-db1983dc6fc4-kube-api-access-jbvcb\") pod \"service-ca-operator-7695fc6c8c-q2g8q\" (UID: \"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:30.831193 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.831174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-bound-sa-token\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:30.831763 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.831743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnpc\" (UniqueName: \"kubernetes.io/projected/c78ebcf5-0805-48e9-b946-4c759026f110-kube-api-access-flnpc\") pod \"insights-operator-68d9dd94cf-6z66z\" (UID: \"c78ebcf5-0805-48e9-b946-4c759026f110\") " pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:30.831763 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.831755 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jbf7\" (UniqueName: \"kubernetes.io/projected/dc87cee2-2ef1-4d66-977f-47f83f7cc504-kube-api-access-6jbf7\") pod \"console-operator-7866497494-fmwwd\" (UID: \"dc87cee2-2ef1-4d66-977f-47f83f7cc504\") " pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.833585 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.833569 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 22:09:30.833685 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.833664 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 22:09:30.833795 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.833779 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nwz7d\"" Jul 14 22:09:30.854550 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.854535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt" Jul 14 22:09:30.871168 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.871148 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:30.887480 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.887456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2" Jul 14 22:09:30.891556 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.891525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" Jul 14 22:09:30.913514 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.913022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0784199-9bd0-4ecf-a5d5-ef31376c45a8-cert\") pod \"ingress-canary-5mbjm\" (UID: \"d0784199-9bd0-4ecf-a5d5-ef31376c45a8\") " pod="openshift-ingress-canary/ingress-canary-5mbjm" Jul 14 22:09:30.913514 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.913140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc2b4f2-7551-4c24-95c1-21388f68f67b-tmp-dir\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.913514 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.913202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qktr4\" (UniqueName: \"kubernetes.io/projected/9dc2b4f2-7551-4c24-95c1-21388f68f67b-kube-api-access-qktr4\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.913514 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.913253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dzg\" (UniqueName: \"kubernetes.io/projected/d0784199-9bd0-4ecf-a5d5-ef31376c45a8-kube-api-access-b6dzg\") pod \"ingress-canary-5mbjm\" (UID: \"d0784199-9bd0-4ecf-a5d5-ef31376c45a8\") " pod="openshift-ingress-canary/ingress-canary-5mbjm" Jul 14 22:09:30.913514 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.913369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.913514 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.913426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc2b4f2-7551-4c24-95c1-21388f68f67b-config-volume\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.914652 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.914051 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc2b4f2-7551-4c24-95c1-21388f68f67b-config-volume\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.914652 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.914474 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc2b4f2-7551-4c24-95c1-21388f68f67b-tmp-dir\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.914824 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.914713 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 22:09:30.915065 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:30.915044 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls podName:9dc2b4f2-7551-4c24-95c1-21388f68f67b nodeName:}" failed. No retries permitted until 2026-07-14 22:09:31.414887424 +0000 UTC m=+37.227519043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls") pod "dns-default-55p9j" (UID: "9dc2b4f2-7551-4c24-95c1-21388f68f67b") : secret "dns-default-metrics-tls" not found Jul 14 22:09:30.921144 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.921028 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0784199-9bd0-4ecf-a5d5-ef31376c45a8-cert\") pod \"ingress-canary-5mbjm\" (UID: \"d0784199-9bd0-4ecf-a5d5-ef31376c45a8\") " pod="openshift-ingress-canary/ingress-canary-5mbjm" Jul 14 22:09:30.925371 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.925220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:30.926590 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.926563 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktr4\" (UniqueName: \"kubernetes.io/projected/9dc2b4f2-7551-4c24-95c1-21388f68f67b-kube-api-access-qktr4\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:30.927953 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.927895 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dzg\" (UniqueName: \"kubernetes.io/projected/d0784199-9bd0-4ecf-a5d5-ef31376c45a8-kube-api-access-b6dzg\") pod \"ingress-canary-5mbjm\" (UID: \"d0784199-9bd0-4ecf-a5d5-ef31376c45a8\") " pod="openshift-ingress-canary/ingress-canary-5mbjm" Jul 14 22:09:30.996252 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.995965 2578 generic.go:356] "Generic (PLEG): container finished" podID="aee8d65f-586f-4879-a62f-69223df24c06" containerID="4965d149c8d3396b715ad022cb7540a105e00de315eb6bfc4f8cc8e31b4e1eea" exitCode=0 Jul 14 22:09:30.996252 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:30.996014 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" event={"ID":"aee8d65f-586f-4879-a62f-69223df24c06","Type":"ContainerDied","Data":"4965d149c8d3396b715ad022cb7540a105e00de315eb6bfc4f8cc8e31b4e1eea"} Jul 14 22:09:31.007362 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.007330 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5mbjm" Jul 14 22:09:31.043182 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.041714 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt"] Jul 14 22:09:31.047827 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:31.047782 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb093ad9b_7ff0_458a_8208_6356b7a1332b.slice/crio-1adb984b81287acb622d40b36fdc62591087fa0752b3d3fae5e46f3814032249 WatchSource:0}: Error finding container 1adb984b81287acb622d40b36fdc62591087fa0752b3d3fae5e46f3814032249: Status 404 returned error can't find the container with id 1adb984b81287acb622d40b36fdc62591087fa0752b3d3fae5e46f3814032249 Jul 14 22:09:31.066617 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.066545 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-fmwwd"] Jul 14 22:09:31.073545 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:31.073472 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc87cee2_2ef1_4d66_977f_47f83f7cc504.slice/crio-16ca147bd6ba19bd66de5b652add832f6b576fc170097a3c7641b78869d3028c WatchSource:0}: Error finding container 16ca147bd6ba19bd66de5b652add832f6b576fc170097a3c7641b78869d3028c: Status 404 returned error can't find the container with id 16ca147bd6ba19bd66de5b652add832f6b576fc170097a3c7641b78869d3028c Jul 14 22:09:31.089739 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.089558 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq"] Jul 14 22:09:31.094923 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:31.094726 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fe832f_c448_4835_bdd6_2c03d94edc5f.slice/crio-a4f7f69137214c87ec25c4774dcbc80cfd64f897d1f635ceffb11db4f2c74fbc WatchSource:0}: Error finding container a4f7f69137214c87ec25c4774dcbc80cfd64f897d1f635ceffb11db4f2c74fbc: Status 404 returned error can't find the container with id a4f7f69137214c87ec25c4774dcbc80cfd64f897d1f635ceffb11db4f2c74fbc Jul 14 22:09:31.095555 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.095418 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:31.097098 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.097070 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2"] Jul 14 22:09:31.100939 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:31.100913 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4aa4416_888a_403e_9a78_07bd8ec23af4.slice/crio-b0328d262b10324d3ef8cc4f2d9323397c2c1ffa52c82504a8e0267ac9f3124d WatchSource:0}: Error finding container b0328d262b10324d3ef8cc4f2d9323397c2c1ffa52c82504a8e0267ac9f3124d: Status 404 returned error can't find the container with id b0328d262b10324d3ef8cc4f2d9323397c2c1ffa52c82504a8e0267ac9f3124d Jul 14 22:09:31.121791 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.121764 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" Jul 14 22:09:31.126167 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.126145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" Jul 14 22:09:31.159039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.159016 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5mbjm"] Jul 14 22:09:31.184213 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:31.184054 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0784199_9bd0_4ecf_a5d5_ef31376c45a8.slice/crio-0a4ee98de6cf191119132cbc53c600bc85f82f6c139112d6caece14940631d17 WatchSource:0}: Error finding container 0a4ee98de6cf191119132cbc53c600bc85f82f6c139112d6caece14940631d17: Status 404 returned error can't find the container with id 0a4ee98de6cf191119132cbc53c600bc85f82f6c139112d6caece14940631d17 Jul 14 22:09:31.256459 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.256429 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:31.284832 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.284749 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q"] Jul 14 22:09:31.286895 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:31.286874 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca2dc15_2a22_4ed0_b950_db1983dc6fc4.slice/crio-781f766cfddb0f54c3fd390064a1aea610ccdac79c271c06abc8395deabc19fc WatchSource:0}: Error finding container 781f766cfddb0f54c3fd390064a1aea610ccdac79c271c06abc8395deabc19fc: Status 404 returned error can't find the container with id 781f766cfddb0f54c3fd390064a1aea610ccdac79c271c06abc8395deabc19fc Jul 14 22:09:31.294436 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.294413 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-6z66z"] Jul 14 22:09:31.296841 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:31.296822 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78ebcf5_0805_48e9_b946_4c759026f110.slice/crio-7a622c3af85557d10f5480cbe045bd7881f78b0680a77aab83efad008bb250a0 WatchSource:0}: Error finding container 7a622c3af85557d10f5480cbe045bd7881f78b0680a77aab83efad008bb250a0: Status 404 returned error can't find the container with id 7a622c3af85557d10f5480cbe045bd7881f78b0680a77aab83efad008bb250a0 Jul 14 22:09:31.316681 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.316651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:31.316769 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.316695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:31.316769 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.316730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:31.316769 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.316766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:31.316903 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.316788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:31.316903 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.316787 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:31.316903 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.316865 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 22:09:31.316903 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.316862 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 22:09:31.317048 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.316910 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 22:09:31.317048 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.316927 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv: secret "image-registry-tls" not found Jul 14 22:09:31.317048 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.316898 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:32.316883201 +0000 UTC m=+38.129514819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : configmap references non-existent config key: service-ca.crt Jul 14 22:09:31.317048 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.316972 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls podName:92e2a1a7-c30d-4973-a60a-a21ac171201a nodeName:}" failed. No retries permitted until 2026-07-14 22:09:32.316960059 +0000 UTC m=+38.129591678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-w2jcz" (UID: "92e2a1a7-c30d-4973-a60a-a21ac171201a") : secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:31.317048 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.316983 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls podName:786291b9-f5ed-48f9-84af-097f93692bfe nodeName:}" failed. No retries permitted until 2026-07-14 22:09:32.316976283 +0000 UTC m=+38.129607900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls") pod "cluster-samples-operator-65db858894-f9995" (UID: "786291b9-f5ed-48f9-84af-097f93692bfe") : secret "samples-operator-tls" not found Jul 14 22:09:31.317048 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.316995 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:32.316987347 +0000 UTC m=+38.129618966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : secret "router-metrics-certs-default" not found Jul 14 22:09:31.317048 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.317008 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls podName:6455ccbb-3529-44c1-9c6d-ac80be15648f nodeName:}" failed. No retries permitted until 2026-07-14 22:09:32.317001366 +0000 UTC m=+38.129632984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls") pod "image-registry-7c5bd5fbf5-xh6bv" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f") : secret "image-registry-tls" not found Jul 14 22:09:31.356338 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.356310 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:31.418218 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.418193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:31.418370 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.418305 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 22:09:31.418370 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:31.418352 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls podName:9dc2b4f2-7551-4c24-95c1-21388f68f67b nodeName:}" failed. No retries permitted until 2026-07-14 22:09:32.418337796 +0000 UTC m=+38.230969413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls") pod "dns-default-55p9j" (UID: "9dc2b4f2-7551-4c24-95c1-21388f68f67b") : secret "dns-default-metrics-tls" not found Jul 14 22:09:31.654302 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.654105 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:31.711198 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.711091 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:31.831461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.831434 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:09:31.834782 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:31.834572 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcczq\"" Jul 14 22:09:32.008044 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.007904 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-fmwwd" event={"ID":"dc87cee2-2ef1-4d66-977f-47f83f7cc504","Type":"ContainerStarted","Data":"16ca147bd6ba19bd66de5b652add832f6b576fc170097a3c7641b78869d3028c"} Jul 14 22:09:32.009744 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.009668 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5mbjm" event={"ID":"d0784199-9bd0-4ecf-a5d5-ef31376c45a8","Type":"ContainerStarted","Data":"0a4ee98de6cf191119132cbc53c600bc85f82f6c139112d6caece14940631d17"} Jul 14 22:09:32.012314 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.012282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" event={"ID":"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4","Type":"ContainerStarted","Data":"781f766cfddb0f54c3fd390064a1aea610ccdac79c271c06abc8395deabc19fc"} Jul 14 22:09:32.015642 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.015617 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt" event={"ID":"b093ad9b-7ff0-458a-8208-6356b7a1332b","Type":"ContainerStarted","Data":"1adb984b81287acb622d40b36fdc62591087fa0752b3d3fae5e46f3814032249"} Jul 14 22:09:32.025499 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.025473 2578 generic.go:356] "Generic (PLEG): container finished" podID="aee8d65f-586f-4879-a62f-69223df24c06" containerID="49d4d1c78784031d00f7c7be6964fb4673ddbc70a8f8b728ac0482d6a42d4a91" exitCode=0 Jul 14 22:09:32.025601 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.025552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" event={"ID":"aee8d65f-586f-4879-a62f-69223df24c06","Type":"ContainerDied","Data":"49d4d1c78784031d00f7c7be6964fb4673ddbc70a8f8b728ac0482d6a42d4a91"} Jul 14 22:09:32.035665 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.035589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" event={"ID":"c78ebcf5-0805-48e9-b946-4c759026f110","Type":"ContainerStarted","Data":"7a622c3af85557d10f5480cbe045bd7881f78b0680a77aab83efad008bb250a0"} Jul 14 22:09:32.039603 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.039545 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" event={"ID":"95fe832f-c448-4835-bdd6-2c03d94edc5f","Type":"ContainerStarted","Data":"a4f7f69137214c87ec25c4774dcbc80cfd64f897d1f635ceffb11db4f2c74fbc"} Jul 14 22:09:32.041252 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.041216 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2" event={"ID":"f4aa4416-888a-403e-9a78-07bd8ec23af4","Type":"ContainerStarted","Data":"b0328d262b10324d3ef8cc4f2d9323397c2c1ffa52c82504a8e0267ac9f3124d"} Jul 14 22:09:32.326747 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.326711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:32.326911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.326806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:32.326911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.326876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:32.327041 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.326914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:32.327041 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.326968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:32.327139 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327117 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:32.327189 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327181 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls podName:92e2a1a7-c30d-4973-a60a-a21ac171201a nodeName:}" failed. No retries permitted until 2026-07-14 22:09:34.327161802 +0000 UTC m=+40.139793421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-w2jcz" (UID: "92e2a1a7-c30d-4973-a60a-a21ac171201a") : secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:32.328930 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327605 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 22:09:32.328930 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327658 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls podName:786291b9-f5ed-48f9-84af-097f93692bfe nodeName:}" failed. No retries permitted until 2026-07-14 22:09:34.327642478 +0000 UTC m=+40.140274097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls") pod "cluster-samples-operator-65db858894-f9995" (UID: "786291b9-f5ed-48f9-84af-097f93692bfe") : secret "samples-operator-tls" not found Jul 14 22:09:32.328930 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327714 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 22:09:32.328930 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327746 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:34.327736174 +0000 UTC m=+40.140367794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : secret "router-metrics-certs-default" not found Jul 14 22:09:32.328930 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327816 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:34.327797654 +0000 UTC m=+40.140429275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : configmap references non-existent config key: service-ca.crt Jul 14 22:09:32.328930 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327888 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 22:09:32.328930 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327899 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv: secret "image-registry-tls" not found Jul 14 22:09:32.328930 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.327928 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls podName:6455ccbb-3529-44c1-9c6d-ac80be15648f nodeName:}" failed. No retries permitted until 2026-07-14 22:09:34.327918035 +0000 UTC m=+40.140549654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls") pod "image-registry-7c5bd5fbf5-xh6bv" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f") : secret "image-registry-tls" not found Jul 14 22:09:32.428332 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:32.428076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:32.428614 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.428596 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 22:09:32.428730 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:32.428661 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls podName:9dc2b4f2-7551-4c24-95c1-21388f68f67b nodeName:}" failed. No retries permitted until 2026-07-14 22:09:34.428641408 +0000 UTC m=+40.241273025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls") pod "dns-default-55p9j" (UID: "9dc2b4f2-7551-4c24-95c1-21388f68f67b") : secret "dns-default-metrics-tls" not found Jul 14 22:09:33.053276 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:33.052507 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" event={"ID":"aee8d65f-586f-4879-a62f-69223df24c06","Type":"ContainerStarted","Data":"a8830e5b7a8ff57e3ff201779e864fd4d0e9349dcabe5cd9f31ade8c5e911893"} Jul 14 22:09:33.325932 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:33.325902 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:33.422495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:33.422120 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:34.058087 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.058040 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt" event={"ID":"b093ad9b-7ff0-458a-8208-6356b7a1332b","Type":"ContainerStarted","Data":"be80404553bf7e80db21819c982c3c296e571cc51deebc61027a302895f8957f"} Jul 14 22:09:34.078720 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.078657 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-ts2bt" podStartSLOduration=17.980003438 podStartE2EDuration="20.078638304s" totalImagesPullingTime="2.098634866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:34.078283007 +0000 UTC m=+39.890914650" watchObservedRunningTime="2026-07-14 22:09:34.078638304 +0000 UTC m=+39.891269945" Jul 14 22:09:34.079094 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.079052 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d6zkw" podStartSLOduration=9.88113463 podStartE2EDuration="39.079037881s" totalImagesPullingTime="29.197903251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:33.095982611 +0000 UTC m=+38.908614252" watchObservedRunningTime="2026-07-14 22:09:34.079037881 +0000 UTC m=+39.891669522" Jul 14 22:09:34.166366 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.166320 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:34.247705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.247624 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:34.348690 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.348613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:34.348690 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.348668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.348725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.348768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348786 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.348800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348866 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls podName:92e2a1a7-c30d-4973-a60a-a21ac171201a nodeName:}" failed. No retries permitted until 2026-07-14 22:09:38.34884324 +0000 UTC m=+44.161474862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-w2jcz" (UID: "92e2a1a7-c30d-4973-a60a-a21ac171201a") : secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348866 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348905 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348916 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:38.348900408 +0000 UTC m=+44.161532042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : configmap references non-existent config key: service-ca.crt Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348865 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348939 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:38.348928845 +0000 UTC m=+44.161560468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : secret "router-metrics-certs-default" not found Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348967 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls podName:786291b9-f5ed-48f9-84af-097f93692bfe nodeName:}" failed. No retries permitted until 2026-07-14 22:09:38.348957341 +0000 UTC m=+44.161588986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls") pod "cluster-samples-operator-65db858894-f9995" (UID: "786291b9-f5ed-48f9-84af-097f93692bfe") : secret "samples-operator-tls" not found Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348919 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv: secret "image-registry-tls" not found Jul 14 22:09:34.348989 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.348992 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls podName:6455ccbb-3529-44c1-9c6d-ac80be15648f nodeName:}" failed. No retries permitted until 2026-07-14 22:09:38.348987304 +0000 UTC m=+44.161618922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls") pod "image-registry-7c5bd5fbf5-xh6bv" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f") : secret "image-registry-tls" not found Jul 14 22:09:34.449309 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.449274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:34.449486 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.449462 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 22:09:34.449550 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:34.449536 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls podName:9dc2b4f2-7551-4c24-95c1-21388f68f67b nodeName:}" failed. No retries permitted until 2026-07-14 22:09:38.449517362 +0000 UTC m=+44.262148980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls") pod "dns-default-55p9j" (UID: "9dc2b4f2-7551-4c24-95c1-21388f68f67b") : secret "dns-default-metrics-tls" not found Jul 14 22:09:34.907493 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.907455 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:34.982085 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:34.982054 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:38.383412 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:38.383374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:38.383434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383537 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383546 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:38.383584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383595 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:46.383578363 +0000 UTC m=+52.196209982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : secret "router-metrics-certs-default" not found Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383671 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls podName:786291b9-f5ed-48f9-84af-097f93692bfe nodeName:}" failed. No retries permitted until 2026-07-14 22:09:46.383649868 +0000 UTC m=+52.196281489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls") pod "cluster-samples-operator-65db858894-f9995" (UID: "786291b9-f5ed-48f9-84af-097f93692bfe") : secret "samples-operator-tls" not found Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383698 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:09:46.38369093 +0000 UTC m=+52.196322548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : configmap references non-existent config key: service-ca.crt Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:38.383739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:38.383781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383873 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383889 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv: secret "image-registry-tls" not found Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383931 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls podName:6455ccbb-3529-44c1-9c6d-ac80be15648f nodeName:}" failed. No retries permitted until 2026-07-14 22:09:46.383917275 +0000 UTC m=+52.196548901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls") pod "image-registry-7c5bd5fbf5-xh6bv" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f") : secret "image-registry-tls" not found Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383877 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:38.384074 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.383981 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls podName:92e2a1a7-c30d-4973-a60a-a21ac171201a nodeName:}" failed. No retries permitted until 2026-07-14 22:09:46.383971514 +0000 UTC m=+52.196603132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-w2jcz" (UID: "92e2a1a7-c30d-4973-a60a-a21ac171201a") : secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:38.484566 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:38.484526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:38.484739 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.484676 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 22:09:38.484805 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:38.484741 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls podName:9dc2b4f2-7551-4c24-95c1-21388f68f67b nodeName:}" failed. No retries permitted until 2026-07-14 22:09:46.484722178 +0000 UTC m=+52.297353810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls") pod "dns-default-55p9j" (UID: "9dc2b4f2-7551-4c24-95c1-21388f68f67b") : secret "dns-default-metrics-tls" not found Jul 14 22:09:39.076039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:39.075999 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5mbjm" event={"ID":"d0784199-9bd0-4ecf-a5d5-ef31376c45a8","Type":"ContainerStarted","Data":"4131448e6fbebc4f65e89cac85de15facd30829c1ec38f4e37a332a84bc87b12"} Jul 14 22:09:39.078370 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:39.078323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" event={"ID":"95fe832f-c448-4835-bdd6-2c03d94edc5f","Type":"ContainerStarted","Data":"cd0c2cf34e545117490c325e398f717d8008da04df97acd09258c4995e7d1de3"} Jul 14 22:09:39.109983 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:39.108999 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5mbjm" podStartSLOduration=1.385185473 podStartE2EDuration="9.108979472s" totalImagesPullingTime="7.723793999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:39.108552008 +0000 UTC m=+44.921183648" watchObservedRunningTime="2026-07-14 22:09:39.108979472 +0000 UTC m=+44.921611114" Jul 14 22:09:40.088940 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.088902 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" event={"ID":"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4","Type":"ContainerStarted","Data":"1b0121ebd73ddee2f446e34cfb4d78a0f35809154dcdd2ddfac46f4a3923cf32"} Jul 14 22:09:40.091277 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.091203 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" event={"ID":"c78ebcf5-0805-48e9-b946-4c759026f110","Type":"ContainerStarted","Data":"2d0710fa4b74a7e07282933f0d3299f62cd214fb094805741043ca43ab6798f5"} Jul 14 22:09:40.093528 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.093501 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2" event={"ID":"f4aa4416-888a-403e-9a78-07bd8ec23af4","Type":"ContainerStarted","Data":"13f78c8a8e770c4ca88a742e46260eacb065eff0a106921ffb1baebfd54ba567"} Jul 14 22:09:40.095574 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.095554 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/0.log" Jul 14 22:09:40.095678 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.095591 2578 generic.go:356] "Generic (PLEG): container finished" podID="dc87cee2-2ef1-4d66-977f-47f83f7cc504" containerID="d39dd95d12935b2613c8ace16cdb229702e314d751ca7679161c0785f7f3ba6c" exitCode=255 Jul 14 22:09:40.095737 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.095672 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-fmwwd" event={"ID":"dc87cee2-2ef1-4d66-977f-47f83f7cc504","Type":"ContainerDied","Data":"d39dd95d12935b2613c8ace16cdb229702e314d751ca7679161c0785f7f3ba6c"} Jul 14 22:09:40.095828 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.095810 2578 scope.go:117] "RemoveContainer" containerID="d39dd95d12935b2613c8ace16cdb229702e314d751ca7679161c0785f7f3ba6c" Jul 14 22:09:40.108314 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.108264 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" podStartSLOduration=18.486972988 podStartE2EDuration="26.10825019s" totalImagesPullingTime="7.621277202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:40.10754775 +0000 UTC m=+45.920179389" watchObservedRunningTime="2026-07-14 22:09:40.10825019 +0000 UTC m=+45.920881825" Jul 14 22:09:40.138047 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.137985 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" podStartSLOduration=18.518940757 podStartE2EDuration="26.137965462s" totalImagesPullingTime="7.619024705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:40.137105708 +0000 UTC m=+45.949737348" watchObservedRunningTime="2026-07-14 22:09:40.137965462 +0000 UTC m=+45.950597104" Jul 14 22:09:40.214448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.214412 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:40.230800 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.230738 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" podStartSLOduration=18.418768906 podStartE2EDuration="26.230719197s" totalImagesPullingTime="7.811950291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:40.230004987 +0000 UTC m=+46.042636626" watchObservedRunningTime="2026-07-14 22:09:40.230719197 +0000 UTC m=+46.043350841" Jul 14 22:09:40.231090 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.231062 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8vrr2" podStartSLOduration=18.426188903 podStartE2EDuration="26.231054216s" totalImagesPullingTime="7.804865313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:40.204126371 +0000 UTC m=+46.016758090" watchObservedRunningTime="2026-07-14 22:09:40.231054216 +0000 UTC m=+46.043685856" Jul 14 22:09:40.234013 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.233988 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh"] Jul 14 22:09:40.253076 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.253048 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh"] Jul 14 22:09:40.253206 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.253184 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh" Jul 14 22:09:40.258098 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.257500 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 14 22:09:40.258098 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.257726 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fn259\"" Jul 14 22:09:40.258098 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.257940 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 14 22:09:40.301929 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.301903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:40.306045 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.306021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhn95\" (UniqueName: \"kubernetes.io/projected/05bb1b72-634d-4d59-a98a-d7c49e292dcf-kube-api-access-jhn95\") pod \"migrator-7f59b54f6c-v94mh\" (UID: \"05bb1b72-634d-4d59-a98a-d7c49e292dcf\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh" Jul 14 22:09:40.399743 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.399699 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:40.406843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.406819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhn95\" (UniqueName: \"kubernetes.io/projected/05bb1b72-634d-4d59-a98a-d7c49e292dcf-kube-api-access-jhn95\") pod \"migrator-7f59b54f6c-v94mh\" (UID: \"05bb1b72-634d-4d59-a98a-d7c49e292dcf\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh" Jul 14 22:09:40.417741 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.417717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhn95\" (UniqueName: \"kubernetes.io/projected/05bb1b72-634d-4d59-a98a-d7c49e292dcf-kube-api-access-jhn95\") pod \"migrator-7f59b54f6c-v94mh\" (UID: \"05bb1b72-634d-4d59-a98a-d7c49e292dcf\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh" Jul 14 22:09:40.564118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.564076 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vqkwk"] Jul 14 22:09:40.569703 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.569672 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh" Jul 14 22:09:40.585105 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.585083 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vqkwk"] Jul 14 22:09:40.585259 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.585214 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.587987 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.587963 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 22:09:40.588117 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.587963 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 22:09:40.588117 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.588029 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z7z4x\"" Jul 14 22:09:40.611810 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.611787 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:40.683019 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.682932 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:40.709358 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.709328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-crio-socket\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.709358 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.709359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-data-volume\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.709540 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.709439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlp6f\" (UniqueName: \"kubernetes.io/projected/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-api-access-tlp6f\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.709540 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.709507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.709540 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.709524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.713150 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.713128 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh"] Jul 14 22:09:40.716268 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:40.716222 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bb1b72_634d_4d59_a98a_d7c49e292dcf.slice/crio-47b005abc6ecc74e60b14b28b30f1ae00551df86caa3fa823f3e46e883599677 WatchSource:0}: Error finding container 47b005abc6ecc74e60b14b28b30f1ae00551df86caa3fa823f3e46e883599677: Status 404 returned error can't find the container with id 47b005abc6ecc74e60b14b28b30f1ae00551df86caa3fa823f3e46e883599677 Jul 14 22:09:40.810598 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.810571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-crio-socket\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.810713 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.810605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-data-volume\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.810800 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.810770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlp6f\" (UniqueName: \"kubernetes.io/projected/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-api-access-tlp6f\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.810856 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.810800 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-crio-socket\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.810856 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.810830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.810856 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.810849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.811006 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:40.810909 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 22:09:40.811006 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:40.810937 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 22:09:40.811006 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:40.810971 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm podName:2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:41.310950482 +0000 UTC m=+47.123582100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqkwk" (UID: "2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338") : configmap "kube-rbac-proxy" not found Jul 14 22:09:40.811006 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:40.810991 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls podName:2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:41.310981526 +0000 UTC m=+47.123613144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqkwk" (UID: "2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338") : secret "insights-runtime-extractor-tls" not found Jul 14 22:09:40.811182 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.811097 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-data-volume\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.825546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.825489 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlp6f\" (UniqueName: \"kubernetes.io/projected/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-api-access-tlp6f\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:40.871794 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.871773 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:40.871891 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:40.871826 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:41.051648 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.051616 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-jl2d5"] Jul 14 22:09:41.066350 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.066322 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:41.090919 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.090862 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-jl2d5"] Jul 14 22:09:41.091256 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.090944 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:41.093801 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.093769 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qddlb\"" Jul 14 22:09:41.093801 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.093799 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 14 22:09:41.093921 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.093905 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 14 22:09:41.100120 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.100102 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/1.log" Jul 14 22:09:41.100479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.100466 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/0.log" Jul 14 22:09:41.100540 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.100498 2578 generic.go:356] "Generic (PLEG): container finished" podID="dc87cee2-2ef1-4d66-977f-47f83f7cc504" containerID="cb4089b5fb0bf9973d0136cafda9762b7d09872cbffa00ecf6cf4e59aad5e2e6" exitCode=255 Jul 14 22:09:41.100598 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.100580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-fmwwd" event={"ID":"dc87cee2-2ef1-4d66-977f-47f83f7cc504","Type":"ContainerDied","Data":"cb4089b5fb0bf9973d0136cafda9762b7d09872cbffa00ecf6cf4e59aad5e2e6"} Jul 14 22:09:41.100650 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.100635 2578 scope.go:117] "RemoveContainer" containerID="d39dd95d12935b2613c8ace16cdb229702e314d751ca7679161c0785f7f3ba6c" Jul 14 22:09:41.100751 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.100733 2578 scope.go:117] "RemoveContainer" containerID="cb4089b5fb0bf9973d0136cafda9762b7d09872cbffa00ecf6cf4e59aad5e2e6" Jul 14 22:09:41.100918 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:41.100896 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-fmwwd_openshift-console-operator(dc87cee2-2ef1-4d66-977f-47f83f7cc504)\"" pod="openshift-console-operator/console-operator-7866497494-fmwwd" podUID="dc87cee2-2ef1-4d66-977f-47f83f7cc504" Jul 14 22:09:41.101722 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.101698 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh" event={"ID":"05bb1b72-634d-4d59-a98a-d7c49e292dcf","Type":"ContainerStarted","Data":"47b005abc6ecc74e60b14b28b30f1ae00551df86caa3fa823f3e46e883599677"} Jul 14 22:09:41.138156 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.138132 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:41.211771 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.211748 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:41.213874 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.213850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5f7a705d-108e-43b9-8c21-1bcf8789c548-nginx-conf\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:41.214166 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.214144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:41.311818 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.311762 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:41.314970 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.314942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:41.315142 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.314998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:41.315142 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.315059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:41.315142 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:41.315079 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 22:09:41.315142 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.315110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5f7a705d-108e-43b9-8c21-1bcf8789c548-nginx-conf\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:41.315142 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:41.315137 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm podName:2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:42.315120904 +0000 UTC m=+48.127752522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqkwk" (UID: "2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338") : configmap "kube-rbac-proxy" not found Jul 14 22:09:41.315411 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:41.315386 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 22:09:41.315484 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:41.315467 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls podName:2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:42.315444349 +0000 UTC m=+48.128075973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqkwk" (UID: "2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338") : secret "insights-runtime-extractor-tls" not found Jul 14 22:09:41.315556 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:41.315388 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 22:09:41.315556 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:41.315541 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert podName:5f7a705d-108e-43b9-8c21-1bcf8789c548 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:41.81552413 +0000 UTC m=+47.628155753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jl2d5" (UID: "5f7a705d-108e-43b9-8c21-1bcf8789c548") : secret "networking-console-plugin-cert" not found Jul 14 22:09:41.315798 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.315772 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5f7a705d-108e-43b9-8c21-1bcf8789c548-nginx-conf\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:41.415061 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.414988 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:41.511937 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.511901 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:41.820094 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:41.820062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:41.820298 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:41.820211 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 22:09:41.820361 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:41.820306 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert podName:5f7a705d-108e-43b9-8c21-1bcf8789c548 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:42.820289629 +0000 UTC m=+48.632921247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jl2d5" (UID: "5f7a705d-108e-43b9-8c21-1bcf8789c548") : secret "networking-console-plugin-cert" not found Jul 14 22:09:42.107101 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.107023 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/1.log" Jul 14 22:09:42.107461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.107347 2578 scope.go:117] "RemoveContainer" containerID="cb4089b5fb0bf9973d0136cafda9762b7d09872cbffa00ecf6cf4e59aad5e2e6" Jul 14 22:09:42.107516 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:42.107497 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-fmwwd_openshift-console-operator(dc87cee2-2ef1-4d66-977f-47f83f7cc504)\"" pod="openshift-console-operator/console-operator-7866497494-fmwwd" podUID="dc87cee2-2ef1-4d66-977f-47f83f7cc504" Jul 14 22:09:42.325043 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.325008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:42.325043 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.325041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:42.325263 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:42.325127 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 22:09:42.325263 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:42.325190 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm podName:2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:44.325172898 +0000 UTC m=+50.137804516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqkwk" (UID: "2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338") : configmap "kube-rbac-proxy" not found Jul 14 22:09:42.325263 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:42.325134 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 22:09:42.325387 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:42.325269 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls podName:2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:44.325257989 +0000 UTC m=+50.137889607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqkwk" (UID: "2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338") : secret "insights-runtime-extractor-tls" not found Jul 14 22:09:42.639598 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.639575 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-tbpwv"] Jul 14 22:09:42.650136 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.650115 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.651286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.651263 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-tbpwv"] Jul 14 22:09:42.652974 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.652952 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jgf8z\"" Jul 14 22:09:42.653105 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.653082 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 14 22:09:42.653224 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.652965 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 14 22:09:42.653224 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.653040 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 14 22:09:42.653343 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.653277 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 14 22:09:42.729168 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.729099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32c88214-0d72-4c12-9a18-df85c5858555-signing-key\") pod \"service-ca-79f68b9f4f-tbpwv\" (UID: \"32c88214-0d72-4c12-9a18-df85c5858555\") " pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.729299 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.729256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32c88214-0d72-4c12-9a18-df85c5858555-signing-cabundle\") pod \"service-ca-79f68b9f4f-tbpwv\" (UID: \"32c88214-0d72-4c12-9a18-df85c5858555\") " pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.729299 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.729293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grx4\" (UniqueName: \"kubernetes.io/projected/32c88214-0d72-4c12-9a18-df85c5858555-kube-api-access-9grx4\") pod \"service-ca-79f68b9f4f-tbpwv\" (UID: \"32c88214-0d72-4c12-9a18-df85c5858555\") " pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.830009 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.829975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grx4\" (UniqueName: \"kubernetes.io/projected/32c88214-0d72-4c12-9a18-df85c5858555-kube-api-access-9grx4\") pod \"service-ca-79f68b9f4f-tbpwv\" (UID: \"32c88214-0d72-4c12-9a18-df85c5858555\") " pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.830173 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.830067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32c88214-0d72-4c12-9a18-df85c5858555-signing-key\") pod \"service-ca-79f68b9f4f-tbpwv\" (UID: \"32c88214-0d72-4c12-9a18-df85c5858555\") " pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.830267 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.830211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:42.830323 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.830272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32c88214-0d72-4c12-9a18-df85c5858555-signing-cabundle\") pod \"service-ca-79f68b9f4f-tbpwv\" (UID: \"32c88214-0d72-4c12-9a18-df85c5858555\") " pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.830370 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:42.830347 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 22:09:42.830439 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:42.830424 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert podName:5f7a705d-108e-43b9-8c21-1bcf8789c548 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:44.830403577 +0000 UTC m=+50.643035195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jl2d5" (UID: "5f7a705d-108e-43b9-8c21-1bcf8789c548") : secret "networking-console-plugin-cert" not found Jul 14 22:09:42.830914 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.830892 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32c88214-0d72-4c12-9a18-df85c5858555-signing-cabundle\") pod \"service-ca-79f68b9f4f-tbpwv\" (UID: \"32c88214-0d72-4c12-9a18-df85c5858555\") " pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.832428 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.832410 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32c88214-0d72-4c12-9a18-df85c5858555-signing-key\") pod \"service-ca-79f68b9f4f-tbpwv\" (UID: \"32c88214-0d72-4c12-9a18-df85c5858555\") " pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.838634 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.838615 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grx4\" (UniqueName: \"kubernetes.io/projected/32c88214-0d72-4c12-9a18-df85c5858555-kube-api-access-9grx4\") pod \"service-ca-79f68b9f4f-tbpwv\" (UID: \"32c88214-0d72-4c12-9a18-df85c5858555\") " pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.855819 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.855799 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klpfw_4598f53c-2bba-424d-8c59-bec7a4697d98/dns-node-resolver/0.log" Jul 14 22:09:42.943627 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.943598 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc"] Jul 14 22:09:42.960339 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.960316 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" Jul 14 22:09:42.974198 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.974165 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq"] Jul 14 22:09:42.974198 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.974175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:42.977074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.977054 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 22:09:42.977175 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.977062 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 22:09:42.977483 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.977465 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 14 22:09:42.977554 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.977469 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 22:09:42.988118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.988099 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn"] Jul 14 22:09:42.988268 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.988252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" Jul 14 22:09:42.991046 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.991027 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 14 22:09:42.991046 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:42.991043 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5g8xz\"" Jul 14 22:09:43.020612 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.020583 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc"] Jul 14 22:09:43.020762 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.020636 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq"] Jul 14 22:09:43.020762 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.020654 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn"] Jul 14 22:09:43.020762 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.020756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.023814 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.023792 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 14 22:09:43.023939 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.023828 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 14 22:09:43.023939 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.023903 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 14 22:09:43.024024 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.023941 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 14 22:09:43.032135 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.032109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1e939da-3b79-456f-811b-1c0a0294961d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54cbb576fb-9jfdq\" (UID: \"b1e939da-3b79-456f-811b-1c0a0294961d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" Jul 14 22:09:43.032229 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.032216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d262888-0592-40bb-ad3b-0b4c392a5096-tmp\") pod \"klusterlet-addon-workmgr-99c869df9-zwnzc\" (UID: \"8d262888-0592-40bb-ad3b-0b4c392a5096\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.032321 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.032301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8d262888-0592-40bb-ad3b-0b4c392a5096-klusterlet-config\") pod \"klusterlet-addon-workmgr-99c869df9-zwnzc\" (UID: \"8d262888-0592-40bb-ad3b-0b4c392a5096\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.032389 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.032370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92nm\" (UniqueName: \"kubernetes.io/projected/8d262888-0592-40bb-ad3b-0b4c392a5096-kube-api-access-c92nm\") pod \"klusterlet-addon-workmgr-99c869df9-zwnzc\" (UID: \"8d262888-0592-40bb-ad3b-0b4c392a5096\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.032447 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.032431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsx6m\" (UniqueName: \"kubernetes.io/projected/b1e939da-3b79-456f-811b-1c0a0294961d-kube-api-access-zsx6m\") pod \"managed-serviceaccount-addon-agent-54cbb576fb-9jfdq\" (UID: \"b1e939da-3b79-456f-811b-1c0a0294961d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" Jul 14 22:09:43.107677 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.107638 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-tbpwv"] Jul 14 22:09:43.110341 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:43.110309 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c88214_0d72_4c12_9a18_df85c5858555.slice/crio-733ded39dde616985f0de044c3002a76d058cb1a5679322136b7a8060c0310b8 WatchSource:0}: Error finding container 733ded39dde616985f0de044c3002a76d058cb1a5679322136b7a8060c0310b8: Status 404 returned error can't find the container with id 733ded39dde616985f0de044c3002a76d058cb1a5679322136b7a8060c0310b8 Jul 14 22:09:43.111775 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.111748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh" event={"ID":"05bb1b72-634d-4d59-a98a-d7c49e292dcf","Type":"ContainerStarted","Data":"0697844a9dd47b3b36bd48fe4c82c0a2926bc8b84069789fced209c5af172b1a"} Jul 14 22:09:43.111864 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.111795 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh" event={"ID":"05bb1b72-634d-4d59-a98a-d7c49e292dcf","Type":"ContainerStarted","Data":"ca9cb590777166f0214de57b4e971eeec1f27c9e077c5a1d09192afa10731e3a"} Jul 14 22:09:43.132646 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.132603 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-v94mh" podStartSLOduration=1.438619509 podStartE2EDuration="3.132587184s" totalImagesPullingTime="1.693967675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:43.13168631 +0000 UTC m=+48.944317951" watchObservedRunningTime="2026-07-14 22:09:43.132587184 +0000 UTC m=+48.945218824" Jul 14 22:09:43.132969 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.132951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-hub\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.133029 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d262888-0592-40bb-ad3b-0b4c392a5096-tmp\") pod \"klusterlet-addon-workmgr-99c869df9-zwnzc\" (UID: \"8d262888-0592-40bb-ad3b-0b4c392a5096\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.133063 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.133113 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-ca\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.133113 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpwc\" (UniqueName: \"kubernetes.io/projected/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-kube-api-access-7tpwc\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.133190 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8d262888-0592-40bb-ad3b-0b4c392a5096-klusterlet-config\") pod \"klusterlet-addon-workmgr-99c869df9-zwnzc\" (UID: \"8d262888-0592-40bb-ad3b-0b4c392a5096\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.133279 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.133314 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.133396 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92nm\" (UniqueName: \"kubernetes.io/projected/8d262888-0592-40bb-ad3b-0b4c392a5096-kube-api-access-c92nm\") pod \"klusterlet-addon-workmgr-99c869df9-zwnzc\" (UID: \"8d262888-0592-40bb-ad3b-0b4c392a5096\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.133396 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133365 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d262888-0592-40bb-ad3b-0b4c392a5096-tmp\") pod \"klusterlet-addon-workmgr-99c869df9-zwnzc\" (UID: \"8d262888-0592-40bb-ad3b-0b4c392a5096\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.133466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsx6m\" (UniqueName: \"kubernetes.io/projected/b1e939da-3b79-456f-811b-1c0a0294961d-kube-api-access-zsx6m\") pod \"managed-serviceaccount-addon-agent-54cbb576fb-9jfdq\" (UID: \"b1e939da-3b79-456f-811b-1c0a0294961d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" Jul 14 22:09:43.133466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.133459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1e939da-3b79-456f-811b-1c0a0294961d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54cbb576fb-9jfdq\" (UID: \"b1e939da-3b79-456f-811b-1c0a0294961d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" Jul 14 22:09:43.135607 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.135588 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8d262888-0592-40bb-ad3b-0b4c392a5096-klusterlet-config\") pod \"klusterlet-addon-workmgr-99c869df9-zwnzc\" (UID: \"8d262888-0592-40bb-ad3b-0b4c392a5096\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.135696 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.135681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1e939da-3b79-456f-811b-1c0a0294961d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54cbb576fb-9jfdq\" (UID: \"b1e939da-3b79-456f-811b-1c0a0294961d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" Jul 14 22:09:43.144612 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.144593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsx6m\" (UniqueName: \"kubernetes.io/projected/b1e939da-3b79-456f-811b-1c0a0294961d-kube-api-access-zsx6m\") pod \"managed-serviceaccount-addon-agent-54cbb576fb-9jfdq\" (UID: \"b1e939da-3b79-456f-811b-1c0a0294961d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" Jul 14 22:09:43.144903 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.144887 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92nm\" (UniqueName: \"kubernetes.io/projected/8d262888-0592-40bb-ad3b-0b4c392a5096-kube-api-access-c92nm\") pod \"klusterlet-addon-workmgr-99c869df9-zwnzc\" (UID: \"8d262888-0592-40bb-ad3b-0b4c392a5096\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.233827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.233770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.233827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.233821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-ca\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.233981 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.233839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpwc\" (UniqueName: \"kubernetes.io/projected/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-kube-api-access-7tpwc\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.233981 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.233929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.233981 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.233972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.234305 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.234283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-hub\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.234468 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.234446 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.236412 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.236362 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.236509 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.236469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.236509 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.236496 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-hub\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.236598 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.236512 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-ca\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.242437 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.242411 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpwc\" (UniqueName: \"kubernetes.io/projected/a2662d1c-3d9e-467e-b8c4-f3fe08cf911b-kube-api-access-7tpwc\") pod \"cluster-proxy-proxy-agent-6854564b8b-zj8tn\" (UID: \"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.286162 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.286130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:43.311412 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.311389 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" Jul 14 22:09:43.329982 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.329963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:09:43.467110 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.467074 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc"] Jul 14 22:09:43.471791 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:43.471753 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d262888_0592_40bb_ad3b_0b4c392a5096.slice/crio-de5a6da9dc91e5387f4744faf3634a4bebf9a4da8fa911573252fe42f02aceda WatchSource:0}: Error finding container de5a6da9dc91e5387f4744faf3634a4bebf9a4da8fa911573252fe42f02aceda: Status 404 returned error can't find the container with id de5a6da9dc91e5387f4744faf3634a4bebf9a4da8fa911573252fe42f02aceda Jul 14 22:09:43.483118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.482976 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq"] Jul 14 22:09:43.487199 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:43.487109 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1e939da_3b79_456f_811b_1c0a0294961d.slice/crio-6f80f25539ff02e0ce46cb5bde9bff95388670451718de7c6110c42177616460 WatchSource:0}: Error finding container 6f80f25539ff02e0ce46cb5bde9bff95388670451718de7c6110c42177616460: Status 404 returned error can't find the container with id 6f80f25539ff02e0ce46cb5bde9bff95388670451718de7c6110c42177616460 Jul 14 22:09:43.540942 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.540920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn"] Jul 14 22:09:43.542643 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:43.542620 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2662d1c_3d9e_467e_b8c4_f3fe08cf911b.slice/crio-667c22532b510f20e78e911476d5a6e7483711fd06f744778ea93f0784e54f29 WatchSource:0}: Error finding container 667c22532b510f20e78e911476d5a6e7483711fd06f744778ea93f0784e54f29: Status 404 returned error can't find the container with id 667c22532b510f20e78e911476d5a6e7483711fd06f744778ea93f0784e54f29 Jul 14 22:09:43.855921 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:43.855894 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lrct7_05ccb78a-325c-4951-b04d-d4b614127b78/node-ca/0.log" Jul 14 22:09:44.116889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.116794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" event={"ID":"32c88214-0d72-4c12-9a18-df85c5858555","Type":"ContainerStarted","Data":"94887a6cff52c35129ebf2ca12d4c8231fb4f8e79f626b1a75f0517b608c292d"} Jul 14 22:09:44.116889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.116885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" event={"ID":"32c88214-0d72-4c12-9a18-df85c5858555","Type":"ContainerStarted","Data":"733ded39dde616985f0de044c3002a76d058cb1a5679322136b7a8060c0310b8"} Jul 14 22:09:44.117992 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.117960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" event={"ID":"8d262888-0592-40bb-ad3b-0b4c392a5096","Type":"ContainerStarted","Data":"de5a6da9dc91e5387f4744faf3634a4bebf9a4da8fa911573252fe42f02aceda"} Jul 14 22:09:44.119058 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.119028 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" event={"ID":"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b","Type":"ContainerStarted","Data":"667c22532b510f20e78e911476d5a6e7483711fd06f744778ea93f0784e54f29"} Jul 14 22:09:44.120146 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.120122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" event={"ID":"b1e939da-3b79-456f-811b-1c0a0294961d","Type":"ContainerStarted","Data":"6f80f25539ff02e0ce46cb5bde9bff95388670451718de7c6110c42177616460"} Jul 14 22:09:44.136162 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.136117 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-tbpwv" podStartSLOduration=2.136103563 podStartE2EDuration="2.136103563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:44.134723467 +0000 UTC m=+49.947355107" watchObservedRunningTime="2026-07-14 22:09:44.136103563 +0000 UTC m=+49.948735180" Jul 14 22:09:44.255771 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.255725 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5mbjm_d0784199-9bd0-4ecf-a5d5-ef31376c45a8/serve-healthcheck-canary/0.log" Jul 14 22:09:44.348351 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.348319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:44.348351 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.348357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:44.349313 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:44.348571 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 22:09:44.349313 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:44.348624 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls podName:2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:48.348611181 +0000 UTC m=+54.161242798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqkwk" (UID: "2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338") : secret "insights-runtime-extractor-tls" not found Jul 14 22:09:44.349313 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:44.348943 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 22:09:44.349313 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:44.348992 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm podName:2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:48.348975374 +0000 UTC m=+54.161607006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqkwk" (UID: "2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338") : configmap "kube-rbac-proxy" not found Jul 14 22:09:44.459410 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.459319 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-v94mh_05bb1b72-634d-4d59-a98a-d7c49e292dcf/migrator/0.log" Jul 14 22:09:44.657118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.657028 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-v94mh_05bb1b72-634d-4d59-a98a-d7c49e292dcf/graceful-termination/0.log" Jul 14 22:09:44.858452 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.858303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:44.860720 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:44.858722 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 22:09:44.860720 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:44.858793 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert podName:5f7a705d-108e-43b9-8c21-1bcf8789c548 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:48.858771441 +0000 UTC m=+54.671403061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jl2d5" (UID: "5f7a705d-108e-43b9-8c21-1bcf8789c548") : secret "networking-console-plugin-cert" not found Jul 14 22:09:44.861837 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:44.861788 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-66wfq_95fe832f-c448-4835-bdd6-2c03d94edc5f/kube-storage-version-migrator-operator/0.log" Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:46.384314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:46.384414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:46.384494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:46.384563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:46.384601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.384762 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.384778 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv: secret "image-registry-tls" not found Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.384837 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls podName:6455ccbb-3529-44c1-9c6d-ac80be15648f nodeName:}" failed. No retries permitted until 2026-07-14 22:10:02.384816943 +0000 UTC m=+68.197448566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls") pod "image-registry-7c5bd5fbf5-xh6bv" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f") : secret "image-registry-tls" not found Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.384905 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:46.385134 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.384941 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls podName:92e2a1a7-c30d-4973-a60a-a21ac171201a nodeName:}" failed. No retries permitted until 2026-07-14 22:10:02.384930309 +0000 UTC m=+68.197561931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-w2jcz" (UID: "92e2a1a7-c30d-4973-a60a-a21ac171201a") : secret "cluster-monitoring-operator-tls" not found Jul 14 22:09:46.386068 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.385217 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:10:02.385199315 +0000 UTC m=+68.197830950 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : configmap references non-existent config key: service-ca.crt Jul 14 22:09:46.386068 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.385301 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 22:09:46.386068 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.385459 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs podName:b952986a-e1d3-4ea4-86f0-6411a8a41eed nodeName:}" failed. No retries permitted until 2026-07-14 22:10:02.385440062 +0000 UTC m=+68.198071682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs") pod "router-default-6b494b4dd8-57vzj" (UID: "b952986a-e1d3-4ea4-86f0-6411a8a41eed") : secret "router-metrics-certs-default" not found Jul 14 22:09:46.387689 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:46.387660 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/786291b9-f5ed-48f9-84af-097f93692bfe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-f9995\" (UID: \"786291b9-f5ed-48f9-84af-097f93692bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:46.479338 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:46.479286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" Jul 14 22:09:46.486402 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:46.486330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:09:46.486557 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.486493 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 22:09:46.486621 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:46.486570 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls podName:9dc2b4f2-7551-4c24-95c1-21388f68f67b nodeName:}" failed. No retries permitted until 2026-07-14 22:10:02.486548339 +0000 UTC m=+68.299179985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls") pod "dns-default-55p9j" (UID: "9dc2b4f2-7551-4c24-95c1-21388f68f67b") : secret "dns-default-metrics-tls" not found Jul 14 22:09:48.406468 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:48.406421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:48.406919 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:48.406476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:48.406919 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:48.406603 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 22:09:48.406919 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:48.406685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls podName:2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:56.406662332 +0000 UTC m=+62.219293956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqkwk" (UID: "2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338") : secret "insights-runtime-extractor-tls" not found Jul 14 22:09:48.407097 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:48.407001 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:48.911143 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:48.911107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:48.911411 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:48.911249 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 22:09:48.911411 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:48.911316 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert podName:5f7a705d-108e-43b9-8c21-1bcf8789c548 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:56.911297726 +0000 UTC m=+62.723929348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jl2d5" (UID: "5f7a705d-108e-43b9-8c21-1bcf8789c548") : secret "networking-console-plugin-cert" not found Jul 14 22:09:49.072827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:49.072728 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995"] Jul 14 22:09:49.152226 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:49.152165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" event={"ID":"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b","Type":"ContainerStarted","Data":"3f4a1ad5ac6d14e5c4fe4723362c8074c16ffb2f5d7925220160e8199c7bbfb7"} Jul 14 22:09:49.153787 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:49.153746 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" event={"ID":"b1e939da-3b79-456f-811b-1c0a0294961d","Type":"ContainerStarted","Data":"94506efcf62fee125444b9e010c384dbbcbaded8d82980d1e7a0d68364067587"} Jul 14 22:09:49.171019 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:49.170931 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54cbb576fb-9jfdq" podStartSLOduration=1.7261211680000002 podStartE2EDuration="7.170915726s" totalImagesPullingTime="5.444794558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:49.169901143 +0000 UTC m=+54.982532785" watchObservedRunningTime="2026-07-14 22:09:49.170915726 +0000 UTC m=+54.983547366" Jul 14 22:09:50.871815 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:50.871774 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:50.872328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:50.871835 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:09:50.872328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:50.872188 2578 scope.go:117] "RemoveContainer" containerID="cb4089b5fb0bf9973d0136cafda9762b7d09872cbffa00ecf6cf4e59aad5e2e6" Jul 14 22:09:51.162490 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.162411 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:09:51.162927 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.162911 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/1.log" Jul 14 22:09:51.163004 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.162949 2578 generic.go:356] "Generic (PLEG): container finished" podID="dc87cee2-2ef1-4d66-977f-47f83f7cc504" containerID="0510b5101f53cc8c88c8c49712e04476189f8970e74f1d0d06fd2af3aa3ef90b" exitCode=255 Jul 14 22:09:51.163063 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.163038 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-fmwwd" event={"ID":"dc87cee2-2ef1-4d66-977f-47f83f7cc504","Type":"ContainerDied","Data":"0510b5101f53cc8c88c8c49712e04476189f8970e74f1d0d06fd2af3aa3ef90b"} Jul 14 22:09:51.163120 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.163105 2578 scope.go:117] "RemoveContainer" containerID="cb4089b5fb0bf9973d0136cafda9762b7d09872cbffa00ecf6cf4e59aad5e2e6" Jul 14 22:09:51.163494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.163468 2578 scope.go:117] "RemoveContainer" containerID="0510b5101f53cc8c88c8c49712e04476189f8970e74f1d0d06fd2af3aa3ef90b" Jul 14 22:09:51.163643 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:09:51.163624 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-fmwwd_openshift-console-operator(dc87cee2-2ef1-4d66-977f-47f83f7cc504)\"" pod="openshift-console-operator/console-operator-7866497494-fmwwd" podUID="dc87cee2-2ef1-4d66-977f-47f83f7cc504" Jul 14 22:09:51.165722 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.165559 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" event={"ID":"8d262888-0592-40bb-ad3b-0b4c392a5096","Type":"ContainerStarted","Data":"6f9388d868d11f2979bff6a2c476df282a6464e170d9aa9c7ff2f0039474f428"} Jul 14 22:09:51.165831 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.165787 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:51.167331 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.167278 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" event={"ID":"786291b9-f5ed-48f9-84af-097f93692bfe","Type":"ContainerStarted","Data":"af46adfdf7c987a843823cf1a77b46cfa8332cab89fd4349c404785e3e01cf85"} Jul 14 22:09:51.167587 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.167565 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" Jul 14 22:09:51.169968 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.169942 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" event={"ID":"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b","Type":"ContainerStarted","Data":"ddbed56fc937589744f63c6afce914bb2697316b115a2aeedbc6228918d23fe2"} Jul 14 22:09:51.170066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.169984 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" event={"ID":"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b","Type":"ContainerStarted","Data":"48a40bb687851624f41335972bf5ea8eef708b40b45907420f71ea7dccb5c174"} Jul 14 22:09:51.202533 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.202479 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-99c869df9-zwnzc" podStartSLOduration=2.39498945 podStartE2EDuration="9.202460483s" totalImagesPullingTime="6.807471033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:51.202139593 +0000 UTC m=+57.014771229" watchObservedRunningTime="2026-07-14 22:09:51.202460483 +0000 UTC m=+57.015092120" Jul 14 22:09:51.227538 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:51.227486 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" podStartSLOduration=3.818640641 podStartE2EDuration="9.227469256s" totalImagesPullingTime="5.408828615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:51.225524537 +0000 UTC m=+57.038158076" watchObservedRunningTime="2026-07-14 22:09:51.227469256 +0000 UTC m=+57.040100899" Jul 14 22:09:52.175072 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:52.175049 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:09:52.176874 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:52.176841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" event={"ID":"786291b9-f5ed-48f9-84af-097f93692bfe","Type":"ContainerStarted","Data":"8d458d93c81d68a6e78588291ee9c230296a9f498c934e3d99e33248bfbe6d3b"} Jul 14 22:09:52.177146 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:52.176930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" event={"ID":"786291b9-f5ed-48f9-84af-097f93692bfe","Type":"ContainerStarted","Data":"1f51ce8b2c1cafc8c505980ff77974762c1c8379885e87bd56c7f706f5b005e8"} Jul 14 22:09:52.194207 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:52.194158 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-f9995" podStartSLOduration=36.486222953 podStartE2EDuration="38.194142146s" totalImagesPullingTime="1.707919193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:52.193814762 +0000 UTC m=+58.006446381" watchObservedRunningTime="2026-07-14 22:09:52.194142146 +0000 UTC m=+58.006773787" Jul 14 22:09:53.332020 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:53.331972 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" podUID="a2662d1c-3d9e-467e-b8c4-f3fe08cf911b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 14 22:09:53.973685 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:53.973654 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsgzj" Jul 14 22:09:56.480910 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:56.480878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:56.483098 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:56.483068 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqkwk\" (UID: \"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338\") " pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:56.503979 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:56.503957 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z7z4x\"" Jul 14 22:09:56.511618 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:56.511594 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vqkwk" Jul 14 22:09:56.637736 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:56.637712 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vqkwk"] Jul 14 22:09:56.640005 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:56.639964 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aae1eb9_cd7a_4a3b_bad5_a342dd3c5338.slice/crio-4b464d3f24648eb088dda53957cb7eedf5cc2631e70ec1251a6be46e332195a4 WatchSource:0}: Error finding container 4b464d3f24648eb088dda53957cb7eedf5cc2631e70ec1251a6be46e332195a4: Status 404 returned error can't find the container with id 4b464d3f24648eb088dda53957cb7eedf5cc2631e70ec1251a6be46e332195a4 Jul 14 22:09:56.986556 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:56.986522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:56.988839 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:56.988820 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f7a705d-108e-43b9-8c21-1bcf8789c548-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jl2d5\" (UID: \"5f7a705d-108e-43b9-8c21-1bcf8789c548\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:57.002464 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:57.002442 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qddlb\"" Jul 14 22:09:57.010885 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:57.010869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" Jul 14 22:09:57.134594 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:57.134569 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-jl2d5"] Jul 14 22:09:57.138419 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:09:57.138385 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7a705d_108e_43b9_8c21_1bcf8789c548.slice/crio-d347e9bdfa70f40fd2d5fbfe5024c3e6b4871852dab804035f19db438c7dd2ba WatchSource:0}: Error finding container d347e9bdfa70f40fd2d5fbfe5024c3e6b4871852dab804035f19db438c7dd2ba: Status 404 returned error can't find the container with id d347e9bdfa70f40fd2d5fbfe5024c3e6b4871852dab804035f19db438c7dd2ba Jul 14 22:09:57.193423 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:57.193392 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" event={"ID":"5f7a705d-108e-43b9-8c21-1bcf8789c548","Type":"ContainerStarted","Data":"d347e9bdfa70f40fd2d5fbfe5024c3e6b4871852dab804035f19db438c7dd2ba"} Jul 14 22:09:57.194471 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:09:57.194452 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqkwk" event={"ID":"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338","Type":"ContainerStarted","Data":"4b464d3f24648eb088dda53957cb7eedf5cc2631e70ec1251a6be46e332195a4"} Jul 14 22:10:00.522025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.521999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:10:00.522345 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.522235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:10:00.524837 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.524818 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 22:10:00.524939 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.524878 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 22:10:00.534971 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.534947 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/514eec71-0eb2-4101-be06-0abbad141842-original-pull-secret\") pod \"global-pull-secret-syncer-f2j5k\" (UID: \"514eec71-0eb2-4101-be06-0abbad141842\") " pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:10:00.535210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.535192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210b36c9-4f24-42d9-a8aa-97ff81012c72-metrics-certs\") pod \"network-metrics-daemon-wkv7z\" (UID: \"210b36c9-4f24-42d9-a8aa-97ff81012c72\") " pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:10:00.553127 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.553108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2j5k" Jul 14 22:10:00.559908 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.559886 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nwz7d\"" Jul 14 22:10:00.568315 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.568291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkv7z" Jul 14 22:10:00.623111 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.622822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:10:00.627121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.627067 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh54\" (UniqueName: \"kubernetes.io/projected/b0b82aef-9783-434b-9207-ae510b6dc0eb-kube-api-access-phh54\") pod \"network-check-target-fsbc5\" (UID: \"b0b82aef-9783-434b-9207-ae510b6dc0eb\") " pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:10:00.684254 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.684220 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcczq\"" Jul 14 22:10:00.691735 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.691703 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:10:00.693899 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.693877 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2j5k"] Jul 14 22:10:00.696322 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:10:00.696280 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514eec71_0eb2_4101_be06_0abbad141842.slice/crio-6aceba707fb34523f911c6caf3b7da1beeb164a8288b202e2b79531116f8aa15 WatchSource:0}: Error finding container 6aceba707fb34523f911c6caf3b7da1beeb164a8288b202e2b79531116f8aa15: Status 404 returned error can't find the container with id 6aceba707fb34523f911c6caf3b7da1beeb164a8288b202e2b79531116f8aa15 Jul 14 22:10:00.717494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.717467 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wkv7z"] Jul 14 22:10:00.720335 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:10:00.720310 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210b36c9_4f24_42d9_a8aa_97ff81012c72.slice/crio-3906e15d194890c3151c009099a285811a8534776303c0843eafec326d0c3a02 WatchSource:0}: Error finding container 3906e15d194890c3151c009099a285811a8534776303c0843eafec326d0c3a02: Status 404 returned error can't find the container with id 3906e15d194890c3151c009099a285811a8534776303c0843eafec326d0c3a02 Jul 14 22:10:00.816959 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.816824 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fsbc5"] Jul 14 22:10:00.819315 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:10:00.819288 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b82aef_9783_434b_9207_ae510b6dc0eb.slice/crio-6618b2089f96e1fd705a4727470e9eef89e2f5862dd2088a9239d221526abdda WatchSource:0}: Error finding container 6618b2089f96e1fd705a4727470e9eef89e2f5862dd2088a9239d221526abdda: Status 404 returned error can't find the container with id 6618b2089f96e1fd705a4727470e9eef89e2f5862dd2088a9239d221526abdda Jul 14 22:10:00.871737 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.871714 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:10:00.871819 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.871766 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:10:00.872086 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:00.872070 2578 scope.go:117] "RemoveContainer" containerID="0510b5101f53cc8c88c8c49712e04476189f8970e74f1d0d06fd2af3aa3ef90b" Jul 14 22:10:00.872282 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:10:00.872253 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-fmwwd_openshift-console-operator(dc87cee2-2ef1-4d66-977f-47f83f7cc504)\"" pod="openshift-console-operator/console-operator-7866497494-fmwwd" podUID="dc87cee2-2ef1-4d66-977f-47f83f7cc504" Jul 14 22:10:01.209634 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:01.209597 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqkwk" event={"ID":"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338","Type":"ContainerStarted","Data":"40720996d3246720cdbe0c31b6a096f91824c312d63af386498d9779e3389f53"} Jul 14 22:10:01.210989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:01.210960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fsbc5" event={"ID":"b0b82aef-9783-434b-9207-ae510b6dc0eb","Type":"ContainerStarted","Data":"6d5c2b648dc2afeaa94fc788960962cbc63aa6215c4815be9dcc32d291abc11f"} Jul 14 22:10:01.211103 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:01.211006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fsbc5" event={"ID":"b0b82aef-9783-434b-9207-ae510b6dc0eb","Type":"ContainerStarted","Data":"6618b2089f96e1fd705a4727470e9eef89e2f5862dd2088a9239d221526abdda"} Jul 14 22:10:01.211103 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:01.211034 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:10:01.212101 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:01.212080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2j5k" event={"ID":"514eec71-0eb2-4101-be06-0abbad141842","Type":"ContainerStarted","Data":"6aceba707fb34523f911c6caf3b7da1beeb164a8288b202e2b79531116f8aa15"} Jul 14 22:10:01.215867 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:01.215842 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkv7z" event={"ID":"210b36c9-4f24-42d9-a8aa-97ff81012c72","Type":"ContainerStarted","Data":"3906e15d194890c3151c009099a285811a8534776303c0843eafec326d0c3a02"} Jul 14 22:10:01.217143 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:01.217121 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" event={"ID":"5f7a705d-108e-43b9-8c21-1bcf8789c548","Type":"ContainerStarted","Data":"d91c0f1cb95c6bb50b9e2fc5cf023cbe35030220ed566de8ae49d5a74c7e2946"} Jul 14 22:10:01.228130 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:01.228087 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fsbc5" podStartSLOduration=66.228072805 podStartE2EDuration="1m6.228072805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:01.227685163 +0000 UTC m=+67.040316803" watchObservedRunningTime="2026-07-14 22:10:01.228072805 +0000 UTC m=+67.040704448" Jul 14 22:10:01.244450 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:01.244410 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-jl2d5" podStartSLOduration=16.894943282 podStartE2EDuration="20.244399409s" totalImagesPullingTime="3.349456127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:01.243520975 +0000 UTC m=+67.056152617" watchObservedRunningTime="2026-07-14 22:10:01.244399409 +0000 UTC m=+67.057031027" Jul 14 22:10:02.438338 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.438300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:10:02.438772 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.438348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:10:02.438772 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.438383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:10:02.438772 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.438444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:10:02.439106 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.439081 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b952986a-e1d3-4ea4-86f0-6411a8a41eed-service-ca-bundle\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:10:02.441323 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.441294 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e2a1a7-c30d-4973-a60a-a21ac171201a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-w2jcz\" (UID: \"92e2a1a7-c30d-4973-a60a-a21ac171201a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:10:02.441323 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.441307 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b952986a-e1d3-4ea4-86f0-6411a8a41eed-metrics-certs\") pod \"router-default-6b494b4dd8-57vzj\" (UID: \"b952986a-e1d3-4ea4-86f0-6411a8a41eed\") " pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:10:02.441506 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.441302 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"image-registry-7c5bd5fbf5-xh6bv\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:10:02.539493 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.539416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:10:02.541975 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.541948 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dc2b4f2-7551-4c24-95c1-21388f68f67b-metrics-tls\") pod \"dns-default-55p9j\" (UID: \"9dc2b4f2-7551-4c24-95c1-21388f68f67b\") " pod="openshift-dns/dns-default-55p9j" Jul 14 22:10:02.647454 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.645752 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gv7p9\"" Jul 14 22:10:02.655541 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.655483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:10:02.659559 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.658157 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k2gvh\"" Jul 14 22:10:02.665825 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.665804 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" Jul 14 22:10:02.666298 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.666281 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-29dxq\"" Jul 14 22:10:02.674685 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.674656 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:10:02.784831 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.784614 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-29nlw\"" Jul 14 22:10:02.788583 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.788381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-55p9j" Jul 14 22:10:02.877455 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.877416 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv"] Jul 14 22:10:02.888190 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:10:02.888147 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6455ccbb_3529_44c1_9c6d_ac80be15648f.slice/crio-0503a335763a3fc7105fe849e8ba29a48fa0fbfc75e9b8d353cc71e8b5c89274 WatchSource:0}: Error finding container 0503a335763a3fc7105fe849e8ba29a48fa0fbfc75e9b8d353cc71e8b5c89274: Status 404 returned error can't find the container with id 0503a335763a3fc7105fe849e8ba29a48fa0fbfc75e9b8d353cc71e8b5c89274 Jul 14 22:10:02.896749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.896129 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz"] Jul 14 22:10:02.900203 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:10:02.900171 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e2a1a7_c30d_4973_a60a_a21ac171201a.slice/crio-bd6ad170f201c451a9bae1f16182ed035aa0b6a6a32ff80bf000ac200ee9a459 WatchSource:0}: Error finding container bd6ad170f201c451a9bae1f16182ed035aa0b6a6a32ff80bf000ac200ee9a459: Status 404 returned error can't find the container with id bd6ad170f201c451a9bae1f16182ed035aa0b6a6a32ff80bf000ac200ee9a459 Jul 14 22:10:02.966306 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.966215 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b494b4dd8-57vzj"] Jul 14 22:10:02.974430 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:10:02.974399 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb952986a_e1d3_4ea4_86f0_6411a8a41eed.slice/crio-4a176596cb7ca454bac501efb05b2b6fbc6767d4cc1c444b5ee49a2a49175901 WatchSource:0}: Error finding container 4a176596cb7ca454bac501efb05b2b6fbc6767d4cc1c444b5ee49a2a49175901: Status 404 returned error can't find the container with id 4a176596cb7ca454bac501efb05b2b6fbc6767d4cc1c444b5ee49a2a49175901 Jul 14 22:10:02.997728 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:02.997703 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-55p9j"] Jul 14 22:10:03.002146 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:10:03.002113 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc2b4f2_7551_4c24_95c1_21388f68f67b.slice/crio-48a5a675389437c642c81b93977d0cff1a2b33c0d98d2f544309ce6cc3d9b96f WatchSource:0}: Error finding container 48a5a675389437c642c81b93977d0cff1a2b33c0d98d2f544309ce6cc3d9b96f: Status 404 returned error can't find the container with id 48a5a675389437c642c81b93977d0cff1a2b33c0d98d2f544309ce6cc3d9b96f Jul 14 22:10:03.230406 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.230364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqkwk" event={"ID":"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338","Type":"ContainerStarted","Data":"09ec81eea369247c65b1ae8e5f110296aa5de29bca115cc6bff6f77dd03bf7ba"} Jul 14 22:10:03.232021 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.231986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b494b4dd8-57vzj" event={"ID":"b952986a-e1d3-4ea4-86f0-6411a8a41eed","Type":"ContainerStarted","Data":"3610373f5653463351ec3890ddf802bea0d3437c449e36fc25512d4b1025cf93"} Jul 14 22:10:03.232164 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.232042 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b494b4dd8-57vzj" event={"ID":"b952986a-e1d3-4ea4-86f0-6411a8a41eed","Type":"ContainerStarted","Data":"4a176596cb7ca454bac501efb05b2b6fbc6767d4cc1c444b5ee49a2a49175901"} Jul 14 22:10:03.234726 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.234697 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" event={"ID":"6455ccbb-3529-44c1-9c6d-ac80be15648f","Type":"ContainerStarted","Data":"4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d"} Jul 14 22:10:03.234820 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.234749 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" event={"ID":"6455ccbb-3529-44c1-9c6d-ac80be15648f","Type":"ContainerStarted","Data":"0503a335763a3fc7105fe849e8ba29a48fa0fbfc75e9b8d353cc71e8b5c89274"} Jul 14 22:10:03.235349 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.235329 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:10:03.237842 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.237765 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-55p9j" event={"ID":"9dc2b4f2-7551-4c24-95c1-21388f68f67b","Type":"ContainerStarted","Data":"48a5a675389437c642c81b93977d0cff1a2b33c0d98d2f544309ce6cc3d9b96f"} Jul 14 22:10:03.241194 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.240512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkv7z" event={"ID":"210b36c9-4f24-42d9-a8aa-97ff81012c72","Type":"ContainerStarted","Data":"87174caaa0e728a377eb84ea74e3277a688012fe4c0648c3a2be83e88006680a"} Jul 14 22:10:03.241194 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.240559 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkv7z" event={"ID":"210b36c9-4f24-42d9-a8aa-97ff81012c72","Type":"ContainerStarted","Data":"15ee7a0f2a73326d4327db027314e991f60ca2f9e125a87f2d6bc493f18412a2"} Jul 14 22:10:03.242781 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.242755 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" event={"ID":"92e2a1a7-c30d-4973-a60a-a21ac171201a","Type":"ContainerStarted","Data":"bd6ad170f201c451a9bae1f16182ed035aa0b6a6a32ff80bf000ac200ee9a459"} Jul 14 22:10:03.256438 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.256231 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6b494b4dd8-57vzj" podStartSLOduration=49.256211195 podStartE2EDuration="49.256211195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:03.25484375 +0000 UTC m=+69.067475390" watchObservedRunningTime="2026-07-14 22:10:03.256211195 +0000 UTC m=+69.068842838" Jul 14 22:10:03.284978 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.284917 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wkv7z" podStartSLOduration=66.401984717 podStartE2EDuration="1m8.284897309s" totalImagesPullingTime="1.882912592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:03.284089834 +0000 UTC m=+69.096721466" watchObservedRunningTime="2026-07-14 22:10:03.284897309 +0000 UTC m=+69.097528964" Jul 14 22:10:03.325002 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.324890 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" podStartSLOduration=68.324864596 podStartE2EDuration="1m8.324864596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:03.322696438 +0000 UTC m=+69.135328083" watchObservedRunningTime="2026-07-14 22:10:03.324864596 +0000 UTC m=+69.137496239" Jul 14 22:10:03.331943 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.331562 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" podUID="a2662d1c-3d9e-467e-b8c4-f3fe08cf911b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 14 22:10:03.675684 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.675597 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:10:03.678845 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:03.678817 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:10:04.250440 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:04.249986 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:10:04.251516 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:04.251334 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6b494b4dd8-57vzj" Jul 14 22:10:09.270644 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:09.270606 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2j5k" event={"ID":"514eec71-0eb2-4101-be06-0abbad141842","Type":"ContainerStarted","Data":"5394a0c7226b0a257f43c5bd3ec6837b2e74d45350ed80b6ffeb122424250d33"} Jul 14 22:10:09.272506 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:09.272484 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-55p9j" event={"ID":"9dc2b4f2-7551-4c24-95c1-21388f68f67b","Type":"ContainerStarted","Data":"7e4fa6011b25e1b12e201e4aac7cb0d870728510b920257c514740f931fce883"} Jul 14 22:10:09.272593 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:09.272521 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-55p9j" event={"ID":"9dc2b4f2-7551-4c24-95c1-21388f68f67b","Type":"ContainerStarted","Data":"4a3a42c271d369cc7f0592ba7bcfbd05b0528cf67d084db3d5d1efdbeddb3780"} Jul 14 22:10:09.272593 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:09.272543 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-55p9j" Jul 14 22:10:09.274423 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:09.273982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" event={"ID":"92e2a1a7-c30d-4973-a60a-a21ac171201a","Type":"ContainerStarted","Data":"e81a8eab1826c15948b9abfce33e963ff711b4ab307b61a0a5a0d40207339837"} Jul 14 22:10:09.279386 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:09.276806 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqkwk" event={"ID":"2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338","Type":"ContainerStarted","Data":"97bde6202d54d650d34a78168f058d0dc99a8c4314f4261f2419841dc603c998"} Jul 14 22:10:09.291305 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:09.291261 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f2j5k" podStartSLOduration=66.52790723 podStartE2EDuration="1m14.291234659s" totalImagesPullingTime="7.763327429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:09.289325242 +0000 UTC m=+75.101956878" watchObservedRunningTime="2026-07-14 22:10:09.291234659 +0000 UTC m=+75.103866299" Jul 14 22:10:09.314395 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:09.314347 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-55p9j" podStartSLOduration=33.852915071 podStartE2EDuration="39.314333087s" totalImagesPullingTime="5.461418016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:09.313033178 +0000 UTC m=+75.125664819" watchObservedRunningTime="2026-07-14 22:10:09.314333087 +0000 UTC m=+75.126964727" Jul 14 22:10:09.355730 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:09.355630 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-w2jcz" podStartSLOduration=49.796698618 podStartE2EDuration="55.35561265s" totalImagesPullingTime="5.558914032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:09.330611816 +0000 UTC m=+75.143243457" watchObservedRunningTime="2026-07-14 22:10:09.35561265 +0000 UTC m=+75.168244339" Jul 14 22:10:13.331881 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:13.331846 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" podUID="a2662d1c-3d9e-467e-b8c4-f3fe08cf911b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 14 22:10:13.332350 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:13.331906 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" Jul 14 22:10:13.332406 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:13.332374 2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"ddbed56fc937589744f63c6afce914bb2697316b115a2aeedbc6228918d23fe2"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 14 22:10:13.332450 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:13.332424 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" podUID="a2662d1c-3d9e-467e-b8c4-f3fe08cf911b" containerName="service-proxy" containerID="cri-o://ddbed56fc937589744f63c6afce914bb2697316b115a2aeedbc6228918d23fe2" gracePeriod=30 Jul 14 22:10:13.831466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:13.831441 2578 scope.go:117] "RemoveContainer" containerID="0510b5101f53cc8c88c8c49712e04476189f8970e74f1d0d06fd2af3aa3ef90b" Jul 14 22:10:14.297189 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:14.297112 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:10:14.297360 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:14.297222 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-fmwwd" event={"ID":"dc87cee2-2ef1-4d66-977f-47f83f7cc504","Type":"ContainerStarted","Data":"2910dfe9e1c1e767ee6db602043852fe590abaaeb557b8d677a7eef2a3c4740c"} Jul 14 22:10:14.297621 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:14.297590 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:10:14.299818 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:14.299793 2578 generic.go:356] "Generic (PLEG): container finished" podID="a2662d1c-3d9e-467e-b8c4-f3fe08cf911b" containerID="ddbed56fc937589744f63c6afce914bb2697316b115a2aeedbc6228918d23fe2" exitCode=2 Jul 14 22:10:14.299933 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:14.299832 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" event={"ID":"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b","Type":"ContainerDied","Data":"ddbed56fc937589744f63c6afce914bb2697316b115a2aeedbc6228918d23fe2"} Jul 14 22:10:14.299933 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:14.299862 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6854564b8b-zj8tn" event={"ID":"a2662d1c-3d9e-467e-b8c4-f3fe08cf911b","Type":"ContainerStarted","Data":"fdec85472085a04b6952bfb6ebfa5ab6f696ad62433bc20ed1e15a10c31a5b98"} Jul 14 22:10:14.316223 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:14.316177 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vqkwk" podStartSLOduration=22.677670616 podStartE2EDuration="34.316164066s" totalImagesPullingTime="11.63849345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:09.355470823 +0000 UTC m=+75.168102462" watchObservedRunningTime="2026-07-14 22:10:14.316164066 +0000 UTC m=+80.128795706" Jul 14 22:10:14.316819 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:14.316790 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-fmwwd" podStartSLOduration=52.483342868 podStartE2EDuration="1m0.316780444s" totalImagesPullingTime="7.833437576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:14.315852819 +0000 UTC m=+80.128484460" watchObservedRunningTime="2026-07-14 22:10:14.316780444 +0000 UTC m=+80.129412084" Jul 14 22:10:14.518050 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:14.518019 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-fmwwd" Jul 14 22:10:19.282892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:19.282788 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-55p9j" Jul 14 22:10:20.651564 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.651532 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5btft"] Jul 14 22:10:20.705029 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.704997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.709372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.709181 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 22:10:20.709372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.709268 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xrbwq\"" Jul 14 22:10:20.709372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.709334 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 22:10:20.710506 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.710484 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 22:10:20.711042 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.711019 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 22:10:20.790723 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.790691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-textfile\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.790892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.790739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-tls\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.790892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.790767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.790892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.790796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-wtmp\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.790892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.790836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92c063a1-8397-4b68-9a9f-e985e7df7f7a-root\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.791032 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.790934 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92c063a1-8397-4b68-9a9f-e985e7df7f7a-sys\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.791032 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.790972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92c063a1-8397-4b68-9a9f-e985e7df7f7a-metrics-client-ca\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.791032 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.790998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z74nq\" (UniqueName: \"kubernetes.io/projected/92c063a1-8397-4b68-9a9f-e985e7df7f7a-kube-api-access-z74nq\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.791032 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.791027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892202 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92c063a1-8397-4b68-9a9f-e985e7df7f7a-root\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892388 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892295 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92c063a1-8397-4b68-9a9f-e985e7df7f7a-root\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892388 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92c063a1-8397-4b68-9a9f-e985e7df7f7a-sys\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892510 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92c063a1-8397-4b68-9a9f-e985e7df7f7a-metrics-client-ca\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892510 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892450 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z74nq\" (UniqueName: \"kubernetes.io/projected/92c063a1-8397-4b68-9a9f-e985e7df7f7a-kube-api-access-z74nq\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892510 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892455 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92c063a1-8397-4b68-9a9f-e985e7df7f7a-sys\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892510 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-textfile\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-tls\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.892704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-wtmp\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.893006 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.892831 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-wtmp\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.893061 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.893011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-textfile\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.893114 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.893095 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92c063a1-8397-4b68-9a9f-e985e7df7f7a-metrics-client-ca\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.893167 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.893149 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.895003 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.894982 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.895126 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.895108 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92c063a1-8397-4b68-9a9f-e985e7df7f7a-node-exporter-tls\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:20.926443 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:20.926380 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z74nq\" (UniqueName: \"kubernetes.io/projected/92c063a1-8397-4b68-9a9f-e985e7df7f7a-kube-api-access-z74nq\") pod \"node-exporter-5btft\" (UID: \"92c063a1-8397-4b68-9a9f-e985e7df7f7a\") " pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:21.015216 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:21.015189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5btft" Jul 14 22:10:21.055403 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:10:21.055373 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c063a1_8397_4b68_9a9f_e985e7df7f7a.slice/crio-91efc80cff0b867c39d14e467fdc477032aa83a55e46504503bb38bfc48fa586 WatchSource:0}: Error finding container 91efc80cff0b867c39d14e467fdc477032aa83a55e46504503bb38bfc48fa586: Status 404 returned error can't find the container with id 91efc80cff0b867c39d14e467fdc477032aa83a55e46504503bb38bfc48fa586 Jul 14 22:10:21.329336 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:21.329303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5btft" event={"ID":"92c063a1-8397-4b68-9a9f-e985e7df7f7a","Type":"ContainerStarted","Data":"91efc80cff0b867c39d14e467fdc477032aa83a55e46504503bb38bfc48fa586"} Jul 14 22:10:22.660819 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:22.660790 2578 patch_prober.go:28] interesting pod/image-registry-7c5bd5fbf5-xh6bv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 22:10:22.661108 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:22.660842 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" podUID="6455ccbb-3529-44c1-9c6d-ac80be15648f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:10:23.338704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:23.338670 2578 generic.go:356] "Generic (PLEG): container finished" podID="92c063a1-8397-4b68-9a9f-e985e7df7f7a" containerID="3b4baaac8ee9491a2f0c2b228eeaec9fa5c6de38d5d0aa1421928135a9192886" exitCode=0 Jul 14 22:10:23.338874 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:23.338730 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5btft" event={"ID":"92c063a1-8397-4b68-9a9f-e985e7df7f7a","Type":"ContainerDied","Data":"3b4baaac8ee9491a2f0c2b228eeaec9fa5c6de38d5d0aa1421928135a9192886"} Jul 14 22:10:24.348093 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:24.348063 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5btft" event={"ID":"92c063a1-8397-4b68-9a9f-e985e7df7f7a","Type":"ContainerStarted","Data":"9fe1b66996ea3b7f35f691c3b51820f478e9d668a325c530840a0be19325c641"} Jul 14 22:10:24.348554 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:24.348110 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5btft" event={"ID":"92c063a1-8397-4b68-9a9f-e985e7df7f7a","Type":"ContainerStarted","Data":"34442489bcce37817e3da9c133e945391ad99aeaf8613ebe7f8cd026386efd62"} Jul 14 22:10:24.370154 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:24.370104 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5btft" podStartSLOduration=2.736397157 podStartE2EDuration="4.370090456s" totalImagesPullingTime="1.536102246s" totalInitContainerRuntime="97.591053ms" isStatefulPod=true podCreationTimestamp="2026-07-14 22:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:24.368223435 +0000 UTC m=+90.180855077" watchObservedRunningTime="2026-07-14 22:10:24.370090456 +0000 UTC m=+90.182722097" Jul 14 22:10:25.257394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:25.257365 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:10:29.445805 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:29.445773 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv"] Jul 14 22:10:32.223304 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:32.223269 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fsbc5" Jul 14 22:10:45.430397 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:45.430361 2578 generic.go:356] "Generic (PLEG): container finished" podID="95fe832f-c448-4835-bdd6-2c03d94edc5f" containerID="cd0c2cf34e545117490c325e398f717d8008da04df97acd09258c4995e7d1de3" exitCode=0 Jul 14 22:10:45.430759 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:45.430413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" event={"ID":"95fe832f-c448-4835-bdd6-2c03d94edc5f","Type":"ContainerDied","Data":"cd0c2cf34e545117490c325e398f717d8008da04df97acd09258c4995e7d1de3"} Jul 14 22:10:45.430759 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:45.430730 2578 scope.go:117] "RemoveContainer" containerID="cd0c2cf34e545117490c325e398f717d8008da04df97acd09258c4995e7d1de3" Jul 14 22:10:46.436020 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:46.435987 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-66wfq" event={"ID":"95fe832f-c448-4835-bdd6-2c03d94edc5f","Type":"ContainerStarted","Data":"2edb545eb89ee34c7efddec6260d137b9f22c6975ae78d14726fd00705937845"} Jul 14 22:10:50.452886 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:50.452848 2578 generic.go:356] "Generic (PLEG): container finished" podID="6ca2dc15-2a22-4ed0-b950-db1983dc6fc4" containerID="1b0121ebd73ddee2f446e34cfb4d78a0f35809154dcdd2ddfac46f4a3923cf32" exitCode=0 Jul 14 22:10:50.453325 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:50.452921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" event={"ID":"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4","Type":"ContainerDied","Data":"1b0121ebd73ddee2f446e34cfb4d78a0f35809154dcdd2ddfac46f4a3923cf32"} Jul 14 22:10:50.453325 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:50.453234 2578 scope.go:117] "RemoveContainer" containerID="1b0121ebd73ddee2f446e34cfb4d78a0f35809154dcdd2ddfac46f4a3923cf32" Jul 14 22:10:50.454575 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:50.454543 2578 generic.go:356] "Generic (PLEG): container finished" podID="c78ebcf5-0805-48e9-b946-4c759026f110" containerID="2d0710fa4b74a7e07282933f0d3299f62cd214fb094805741043ca43ab6798f5" exitCode=0 Jul 14 22:10:50.454629 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:50.454606 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" event={"ID":"c78ebcf5-0805-48e9-b946-4c759026f110","Type":"ContainerDied","Data":"2d0710fa4b74a7e07282933f0d3299f62cd214fb094805741043ca43ab6798f5"} Jul 14 22:10:50.454859 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:50.454847 2578 scope.go:117] "RemoveContainer" containerID="2d0710fa4b74a7e07282933f0d3299f62cd214fb094805741043ca43ab6798f5" Jul 14 22:10:51.460368 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:51.460332 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-q2g8q" event={"ID":"6ca2dc15-2a22-4ed0-b950-db1983dc6fc4","Type":"ContainerStarted","Data":"8466e5d321d727788087717a144859011a29ea7151e31967c2ac4f145fe4d862"} Jul 14 22:10:51.462274 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:51.462222 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6z66z" event={"ID":"c78ebcf5-0805-48e9-b946-4c759026f110","Type":"ContainerStarted","Data":"5ede809a80da007edc9b5b6f24d5fa0f35b349758514695bedfda762569fb3dc"} Jul 14 22:10:54.466207 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.466144 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" podUID="6455ccbb-3529-44c1-9c6d-ac80be15648f" containerName="registry" containerID="cri-o://4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d" gracePeriod=30 Jul 14 22:10:54.701000 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.700979 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:10:54.772273 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.772190 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-image-registry-private-configuration\") pod \"6455ccbb-3529-44c1-9c6d-ac80be15648f\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " Jul 14 22:10:54.772273 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.772225 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-certificates\") pod \"6455ccbb-3529-44c1-9c6d-ac80be15648f\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " Jul 14 22:10:54.772273 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.772268 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-installation-pull-secrets\") pod \"6455ccbb-3529-44c1-9c6d-ac80be15648f\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " Jul 14 22:10:54.772516 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.772309 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nv9q\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-kube-api-access-7nv9q\") pod \"6455ccbb-3529-44c1-9c6d-ac80be15648f\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " Jul 14 22:10:54.772516 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.772326 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-trusted-ca\") pod \"6455ccbb-3529-44c1-9c6d-ac80be15648f\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " Jul 14 22:10:54.772516 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.772347 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455ccbb-3529-44c1-9c6d-ac80be15648f-ca-trust-extracted\") pod \"6455ccbb-3529-44c1-9c6d-ac80be15648f\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " Jul 14 22:10:54.772516 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.772384 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") pod \"6455ccbb-3529-44c1-9c6d-ac80be15648f\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " Jul 14 22:10:54.772516 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.772430 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-bound-sa-token\") pod \"6455ccbb-3529-44c1-9c6d-ac80be15648f\" (UID: \"6455ccbb-3529-44c1-9c6d-ac80be15648f\") " Jul 14 22:10:54.772946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.772913 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6455ccbb-3529-44c1-9c6d-ac80be15648f" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:10:54.773072 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.773049 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6455ccbb-3529-44c1-9c6d-ac80be15648f" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:10:54.774851 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.774804 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-kube-api-access-7nv9q" (OuterVolumeSpecName: "kube-api-access-7nv9q") pod "6455ccbb-3529-44c1-9c6d-ac80be15648f" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f"). InnerVolumeSpecName "kube-api-access-7nv9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:10:54.775025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.775002 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6455ccbb-3529-44c1-9c6d-ac80be15648f" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:10:54.775025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.775007 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6455ccbb-3529-44c1-9c6d-ac80be15648f" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:10:54.775151 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.775105 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6455ccbb-3529-44c1-9c6d-ac80be15648f" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:10:54.775299 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.775282 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6455ccbb-3529-44c1-9c6d-ac80be15648f" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:10:54.781043 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.781019 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6455ccbb-3529-44c1-9c6d-ac80be15648f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6455ccbb-3529-44c1-9c6d-ac80be15648f" (UID: "6455ccbb-3529-44c1-9c6d-ac80be15648f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:10:54.873708 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.873680 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-bound-sa-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:10:54.873708 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.873706 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-image-registry-private-configuration\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:10:54.873867 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.873720 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-certificates\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:10:54.873867 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.873734 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455ccbb-3529-44c1-9c6d-ac80be15648f-installation-pull-secrets\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:10:54.873867 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.873746 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nv9q\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-kube-api-access-7nv9q\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:10:54.873867 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.873758 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455ccbb-3529-44c1-9c6d-ac80be15648f-trusted-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:10:54.873867 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.873771 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455ccbb-3529-44c1-9c6d-ac80be15648f-ca-trust-extracted\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:10:54.873867 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:54.873783 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455ccbb-3529-44c1-9c6d-ac80be15648f-registry-tls\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:10:55.478356 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:55.478321 2578 generic.go:356] "Generic (PLEG): container finished" podID="6455ccbb-3529-44c1-9c6d-ac80be15648f" containerID="4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d" exitCode=0 Jul 14 22:10:55.478968 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:55.478369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" event={"ID":"6455ccbb-3529-44c1-9c6d-ac80be15648f","Type":"ContainerDied","Data":"4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d"} Jul 14 22:10:55.478968 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:55.478374 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" Jul 14 22:10:55.478968 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:55.478417 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv" event={"ID":"6455ccbb-3529-44c1-9c6d-ac80be15648f","Type":"ContainerDied","Data":"0503a335763a3fc7105fe849e8ba29a48fa0fbfc75e9b8d353cc71e8b5c89274"} Jul 14 22:10:55.478968 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:55.478448 2578 scope.go:117] "RemoveContainer" containerID="4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d" Jul 14 22:10:55.486626 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:55.486608 2578 scope.go:117] "RemoveContainer" containerID="4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d" Jul 14 22:10:55.486896 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:10:55.486875 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d\": container with ID starting with 4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d not found: ID does not exist" containerID="4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d" Jul 14 22:10:55.486946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:55.486905 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d"} err="failed to get container status \"4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d\": rpc error: code = NotFound desc = could not find container \"4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d\": container with ID starting with 4d13d2cc88f3a7ec77f75c5aec85097c6cd1939470a47bee2985edfe7735d52d not found: ID does not exist" Jul 14 22:10:55.499594 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:55.499573 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv"] Jul 14 22:10:55.505096 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:55.505075 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c5bd5fbf5-xh6bv"] Jul 14 22:10:56.836169 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:10:56.836129 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6455ccbb-3529-44c1-9c6d-ac80be15648f" path="/var/lib/kubelet/pods/6455ccbb-3529-44c1-9c6d-ac80be15648f/volumes" Jul 14 22:13:54.618703 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:13:54.618667 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:13:54.624728 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:13:54.624707 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:13:54.628832 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:13:54.628808 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 22:16:25.944105 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.944018 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl"] Jul 14 22:16:25.944607 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.944450 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6455ccbb-3529-44c1-9c6d-ac80be15648f" containerName="registry" Jul 14 22:16:25.944607 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.944468 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6455ccbb-3529-44c1-9c6d-ac80be15648f" containerName="registry" Jul 14 22:16:25.944607 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.944563 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6455ccbb-3529-44c1-9c6d-ac80be15648f" containerName="registry" Jul 14 22:16:25.947660 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.947638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:25.962622 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.962602 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 22:16:25.962787 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.962767 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 22:16:25.963042 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.963019 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 22:16:25.963111 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.963046 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9kkhx\"" Jul 14 22:16:25.963250 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:25.963220 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 22:16:26.007920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.007890 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl"] Jul 14 22:16:26.128950 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.128922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-tmp\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.129074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.128968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.129074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.128991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.129174 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.129073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-manifests\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.129174 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.129115 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6t4\" (UniqueName: \"kubernetes.io/projected/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-kube-api-access-tc6t4\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.229991 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.229909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.229991 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.229952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.229991 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.229991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-manifests\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.230277 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.230030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6t4\" (UniqueName: \"kubernetes.io/projected/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-kube-api-access-tc6t4\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.230277 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.230062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-tmp\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.230459 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.230437 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-manifests\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.232281 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.232255 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-tmp\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.232517 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.232497 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.232599 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.232558 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.264286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.264265 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6t4\" (UniqueName: \"kubernetes.io/projected/124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8-kube-api-access-tc6t4\") pod \"opendatahub-operator-controller-manager-86c5998494-wqgcl\" (UID: \"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.557808 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.557782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:26.700466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.700433 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl"] Jul 14 22:16:26.704174 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:16:26.704149 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124ab4a8_e642_4e7d_9cdf_e3e9ef6b79e8.slice/crio-82afdff40adddb91907ea90d2d031ea1dc40be29969fe231652e4761138528ac WatchSource:0}: Error finding container 82afdff40adddb91907ea90d2d031ea1dc40be29969fe231652e4761138528ac: Status 404 returned error can't find the container with id 82afdff40adddb91907ea90d2d031ea1dc40be29969fe231652e4761138528ac Jul 14 22:16:26.705935 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.705918 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:16:26.709661 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:26.709638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" event={"ID":"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8","Type":"ContainerStarted","Data":"82afdff40adddb91907ea90d2d031ea1dc40be29969fe231652e4761138528ac"} Jul 14 22:16:30.730257 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:30.730200 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" event={"ID":"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8","Type":"ContainerStarted","Data":"194b2d03bcdacdf56f59869e3aae4ec5c9c9a41c9b4f4219f1008d338e9563fe"} Jul 14 22:16:31.736378 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:31.736347 2578 generic.go:356] "Generic (PLEG): container finished" podID="124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8" containerID="194b2d03bcdacdf56f59869e3aae4ec5c9c9a41c9b4f4219f1008d338e9563fe" exitCode=0 Jul 14 22:16:31.736744 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:31.736444 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" event={"ID":"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8","Type":"ContainerDied","Data":"194b2d03bcdacdf56f59869e3aae4ec5c9c9a41c9b4f4219f1008d338e9563fe"} Jul 14 22:16:32.743093 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:32.743057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" event={"ID":"124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8","Type":"ContainerStarted","Data":"cd8dbf2eab839240c25d60e9726b03e5578178ac36ab10e8c22ede9a22599568"} Jul 14 22:16:32.743480 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:32.743130 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:43.749665 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:43.749633 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" Jul 14 22:16:43.773606 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:43.773558 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-wqgcl" podStartSLOduration=14.232063496 podStartE2EDuration="18.773545524s" totalImagesPullingTime="3.520673481s" totalInitContainerRuntime="1.020808547s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:32.78433597 +0000 UTC m=+458.596967612" watchObservedRunningTime="2026-07-14 22:16:43.773545524 +0000 UTC m=+469.586177164" Jul 14 22:16:46.787598 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.787566 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj"] Jul 14 22:16:46.801490 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.801461 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:46.816473 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.816450 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 22:16:46.817874 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.817854 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6xwfl\"" Jul 14 22:16:46.817983 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.817859 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 22:16:46.851516 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.851493 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj"] Jul 14 22:16:46.885409 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.885385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3-tmp\") pod \"kube-auth-proxy-5ddb5db5fc-nf2xj\" (UID: \"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3\") " pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:46.885552 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.885438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcnwg\" (UniqueName: \"kubernetes.io/projected/92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3-kube-api-access-rcnwg\") pod \"kube-auth-proxy-5ddb5db5fc-nf2xj\" (UID: \"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3\") " pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:46.885552 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.885500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3-tls-certs\") pod \"kube-auth-proxy-5ddb5db5fc-nf2xj\" (UID: \"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3\") " pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:46.986176 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.986145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3-tls-certs\") pod \"kube-auth-proxy-5ddb5db5fc-nf2xj\" (UID: \"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3\") " pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:46.986358 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.986190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3-tmp\") pod \"kube-auth-proxy-5ddb5db5fc-nf2xj\" (UID: \"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3\") " pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:46.986358 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.986221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcnwg\" (UniqueName: \"kubernetes.io/projected/92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3-kube-api-access-rcnwg\") pod \"kube-auth-proxy-5ddb5db5fc-nf2xj\" (UID: \"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3\") " pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:46.988418 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.988397 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3-tmp\") pod \"kube-auth-proxy-5ddb5db5fc-nf2xj\" (UID: \"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3\") " pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:46.988585 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:46.988568 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3-tls-certs\") pod \"kube-auth-proxy-5ddb5db5fc-nf2xj\" (UID: \"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3\") " pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:47.066443 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:47.066414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcnwg\" (UniqueName: \"kubernetes.io/projected/92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3-kube-api-access-rcnwg\") pod \"kube-auth-proxy-5ddb5db5fc-nf2xj\" (UID: \"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3\") " pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:47.111016 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:47.110993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" Jul 14 22:16:47.399757 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:47.399667 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj"] Jul 14 22:16:47.402706 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:16:47.402672 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c9bdcf_d96d_4b3f_be8c_fcf5e86d70e3.slice/crio-72eca07d7993a247649b08dd826c85ccd96af4334a77bc3ff6921ed0427c4780 WatchSource:0}: Error finding container 72eca07d7993a247649b08dd826c85ccd96af4334a77bc3ff6921ed0427c4780: Status 404 returned error can't find the container with id 72eca07d7993a247649b08dd826c85ccd96af4334a77bc3ff6921ed0427c4780 Jul 14 22:16:47.807844 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:47.807812 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" event={"ID":"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3","Type":"ContainerStarted","Data":"72eca07d7993a247649b08dd826c85ccd96af4334a77bc3ff6921ed0427c4780"} Jul 14 22:16:48.579828 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.579796 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6pwzw"] Jul 14 22:16:48.588889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.588862 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:16:48.598350 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.598330 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-srqnp\"" Jul 14 22:16:48.598833 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.598818 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 14 22:16:48.604266 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.604232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6pwzw"] Jul 14 22:16:48.698823 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.698784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0faf2cf-314f-4929-9641-8e500964305a-tls-certs\") pod \"model-serving-api-54d89bd5fc-6pwzw\" (UID: \"c0faf2cf-314f-4929-9641-8e500964305a\") " pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:16:48.698989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.698850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qbt\" (UniqueName: \"kubernetes.io/projected/c0faf2cf-314f-4929-9641-8e500964305a-kube-api-access-p5qbt\") pod \"model-serving-api-54d89bd5fc-6pwzw\" (UID: \"c0faf2cf-314f-4929-9641-8e500964305a\") " pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:16:48.799745 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.799711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qbt\" (UniqueName: \"kubernetes.io/projected/c0faf2cf-314f-4929-9641-8e500964305a-kube-api-access-p5qbt\") pod \"model-serving-api-54d89bd5fc-6pwzw\" (UID: \"c0faf2cf-314f-4929-9641-8e500964305a\") " pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:16:48.799907 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.799783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0faf2cf-314f-4929-9641-8e500964305a-tls-certs\") pod \"model-serving-api-54d89bd5fc-6pwzw\" (UID: \"c0faf2cf-314f-4929-9641-8e500964305a\") " pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:16:48.802197 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.802174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0faf2cf-314f-4929-9641-8e500964305a-tls-certs\") pod \"model-serving-api-54d89bd5fc-6pwzw\" (UID: \"c0faf2cf-314f-4929-9641-8e500964305a\") " pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:16:48.817977 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.817946 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qbt\" (UniqueName: \"kubernetes.io/projected/c0faf2cf-314f-4929-9641-8e500964305a-kube-api-access-p5qbt\") pod \"model-serving-api-54d89bd5fc-6pwzw\" (UID: \"c0faf2cf-314f-4929-9641-8e500964305a\") " pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:16:48.899531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:48.899432 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:16:49.051285 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.051257 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6pwzw"] Jul 14 22:16:49.053682 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:16:49.053654 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0faf2cf_314f_4929_9641_8e500964305a.slice/crio-29243dc2de36ffe4cc71f8411ed8cd8971d2d46d247073ddc535d01275000996 WatchSource:0}: Error finding container 29243dc2de36ffe4cc71f8411ed8cd8971d2d46d247073ddc535d01275000996: Status 404 returned error can't find the container with id 29243dc2de36ffe4cc71f8411ed8cd8971d2d46d247073ddc535d01275000996 Jul 14 22:16:49.558511 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.558481 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-59dph"] Jul 14 22:16:49.562829 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.562813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.567016 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.566990 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 22:16:49.567135 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.566991 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 22:16:49.567527 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.567509 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 22:16:49.570116 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.570101 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-g64zc\"" Jul 14 22:16:49.571162 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.571147 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 22:16:49.591338 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.591315 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-59dph"] Jul 14 22:16:49.607606 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.607583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-cacerts\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.607693 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.607610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.607693 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.607645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w524m\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-kube-api-access-w524m\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.607770 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.607703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.607770 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.607722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.607770 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.607760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-token\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.607859 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.607777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42984c53-43dc-4c20-bfa8-e3382be041c5-local-certs\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.708797 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.708767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-cacerts\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.708797 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.708801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.709021 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.708851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w524m\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-kube-api-access-w524m\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.709021 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.708894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.709021 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.708919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.709021 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.708955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-token\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.709021 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.708985 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42984c53-43dc-4c20-bfa8-e3382be041c5-local-certs\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.709938 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.709904 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.712366 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.712340 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42984c53-43dc-4c20-bfa8-e3382be041c5-local-certs\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.712457 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.712350 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.712457 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.712408 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-cacerts\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.712743 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.712726 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.721053 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.721031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-token\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.721293 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.721273 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w524m\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-kube-api-access-w524m\") pod \"istiod-openshift-gateway-7574459656-59dph\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:49.816340 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.816262 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" event={"ID":"c0faf2cf-314f-4929-9641-8e500964305a","Type":"ContainerStarted","Data":"29243dc2de36ffe4cc71f8411ed8cd8971d2d46d247073ddc535d01275000996"} Jul 14 22:16:49.872659 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:49.872633 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:50.021996 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:50.021772 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-59dph"] Jul 14 22:16:50.025140 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:16:50.025104 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42984c53_43dc_4c20_bfa8_e3382be041c5.slice/crio-23850b42d81fd2e3e174fb16f8bf979f756470e9f6c551b8313d4046b4b31ebc WatchSource:0}: Error finding container 23850b42d81fd2e3e174fb16f8bf979f756470e9f6c551b8313d4046b4b31ebc: Status 404 returned error can't find the container with id 23850b42d81fd2e3e174fb16f8bf979f756470e9f6c551b8313d4046b4b31ebc Jul 14 22:16:50.822913 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:50.822874 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" event={"ID":"42984c53-43dc-4c20-bfa8-e3382be041c5","Type":"ContainerStarted","Data":"23850b42d81fd2e3e174fb16f8bf979f756470e9f6c551b8313d4046b4b31ebc"} Jul 14 22:16:51.830580 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:51.830541 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" event={"ID":"c0faf2cf-314f-4929-9641-8e500964305a","Type":"ContainerStarted","Data":"0e4fbe744ed26a6b1b1e2b5735b5e8ee609726c7638924d4bf2d2f3330d7b515"} Jul 14 22:16:51.831039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:51.830630 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:16:51.832683 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:51.832652 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" event={"ID":"92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3","Type":"ContainerStarted","Data":"b72a15255a44c9645a52cab4fb7528fe4add38befc3bcce513beab192b83a813"} Jul 14 22:16:51.856774 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:51.856703 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" podStartSLOduration=1.457190974 podStartE2EDuration="3.856683141s" totalImagesPullingTime="2.399492167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:51.855169443 +0000 UTC m=+477.667801085" watchObservedRunningTime="2026-07-14 22:16:51.856683141 +0000 UTC m=+477.669314782" Jul 14 22:16:51.878913 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:51.878762 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5ddb5db5fc-nf2xj" podStartSLOduration=1.8277763889999998 podStartE2EDuration="5.878743875s" totalImagesPullingTime="4.050967486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:51.878404383 +0000 UTC m=+477.691036024" watchObservedRunningTime="2026-07-14 22:16:51.878743875 +0000 UTC m=+477.691375516" Jul 14 22:16:53.031874 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:53.031834 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:16:53.032182 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:53.031907 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:16:53.848291 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:53.848228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" event={"ID":"42984c53-43dc-4c20-bfa8-e3382be041c5","Type":"ContainerStarted","Data":"4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39"} Jul 14 22:16:53.848477 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:53.848310 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:53.876587 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:53.876529 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" podStartSLOduration=1.872699249 podStartE2EDuration="4.876514695s" totalImagesPullingTime="3.003815446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:53.87572843 +0000 UTC m=+479.688360071" watchObservedRunningTime="2026-07-14 22:16:53.876514695 +0000 UTC m=+479.689146335" Jul 14 22:16:54.853522 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:54.853489 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-59dph container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:16:54.853891 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:54.853545 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" podUID="42984c53-43dc-4c20-bfa8-e3382be041c5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:16:55.074391 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.074350 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v"] Jul 14 22:16:55.079435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.079417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.083214 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.083186 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6fpkj\"" Jul 14 22:16:55.110536 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.110477 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v"] Jul 14 22:16:55.158391 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.158362 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.158522 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.158399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.158522 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.158426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.158522 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.158449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.158522 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.158504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.158718 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.158560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.158718 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.158600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.158718 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.158624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.158826 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.158712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfnv5\" (UniqueName: \"kubernetes.io/projected/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-kube-api-access-vfnv5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.259521 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.259521 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.259701 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfnv5\" (UniqueName: \"kubernetes.io/projected/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-kube-api-access-vfnv5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.259701 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.259701 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.259701 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.259701 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.259924 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.259924 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.260025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.259982 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.260074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.260046 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.260221 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.260202 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.260318 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.260223 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.260589 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.260564 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.261873 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.261849 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.261992 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.261976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.282131 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.282108 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfnv5\" (UniqueName: \"kubernetes.io/projected/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-kube-api-access-vfnv5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.286181 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.286151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42sb8v\" (UID: \"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.389227 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.389149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:55.577982 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:16:55.577953 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff7a9ed_6dad_4e42_9adb_4f2a04d900ea.slice/crio-ec7798e1271d3023c21cbd7f3721c161f35fb9d85bdccbda53321e6fd6bf5538 WatchSource:0}: Error finding container ec7798e1271d3023c21cbd7f3721c161f35fb9d85bdccbda53321e6fd6bf5538: Status 404 returned error can't find the container with id ec7798e1271d3023c21cbd7f3721c161f35fb9d85bdccbda53321e6fd6bf5538 Jul 14 22:16:55.578254 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.578210 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v"] Jul 14 22:16:55.862719 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:55.862684 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" event={"ID":"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea","Type":"ContainerStarted","Data":"ec7798e1271d3023c21cbd7f3721c161f35fb9d85bdccbda53321e6fd6bf5538"} Jul 14 22:16:57.852860 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:57.852827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:58.158354 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:58.158312 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:16:58.158449 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:58.158388 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:16:58.158449 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:58.158417 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:16:58.878655 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:58.878619 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" event={"ID":"7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea","Type":"ContainerStarted","Data":"4235c14b85c4aa1d940a997c7705713532d16fca118059a217bdb5cf2b4b2eb6"} Jul 14 22:16:58.941969 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:58.941895 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" podStartSLOduration=1.3641251699999999 podStartE2EDuration="3.941871687s" totalImagesPullingTime="2.577746517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:58.939620829 +0000 UTC m=+484.752252471" watchObservedRunningTime="2026-07-14 22:16:58.941871687 +0000 UTC m=+484.754503328" Jul 14 22:16:59.197069 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.196976 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-28rcq"] Jul 14 22:16:59.200232 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.200208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:16:59.250073 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.250046 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dt6pg\"" Jul 14 22:16:59.250195 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.250167 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 14 22:16:59.282653 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.282625 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-28rcq"] Jul 14 22:16:59.296547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.296512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-582jc\" (UniqueName: \"kubernetes.io/projected/22f96003-9eb5-47dd-a1c4-c0a3df528f00-kube-api-access-582jc\") pod \"kserve-controller-manager-d6f4b65b8-28rcq\" (UID: \"22f96003-9eb5-47dd-a1c4-c0a3df528f00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:16:59.296681 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.296559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22f96003-9eb5-47dd-a1c4-c0a3df528f00-cert\") pod \"kserve-controller-manager-d6f4b65b8-28rcq\" (UID: \"22f96003-9eb5-47dd-a1c4-c0a3df528f00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:16:59.389288 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.389260 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:59.393979 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.393955 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:59.397340 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.397316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-582jc\" (UniqueName: \"kubernetes.io/projected/22f96003-9eb5-47dd-a1c4-c0a3df528f00-kube-api-access-582jc\") pod \"kserve-controller-manager-d6f4b65b8-28rcq\" (UID: \"22f96003-9eb5-47dd-a1c4-c0a3df528f00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:16:59.397448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.397347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22f96003-9eb5-47dd-a1c4-c0a3df528f00-cert\") pod \"kserve-controller-manager-d6f4b65b8-28rcq\" (UID: \"22f96003-9eb5-47dd-a1c4-c0a3df528f00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:16:59.399834 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.399808 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22f96003-9eb5-47dd-a1c4-c0a3df528f00-cert\") pod \"kserve-controller-manager-d6f4b65b8-28rcq\" (UID: \"22f96003-9eb5-47dd-a1c4-c0a3df528f00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:16:59.464444 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.464382 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-582jc\" (UniqueName: \"kubernetes.io/projected/22f96003-9eb5-47dd-a1c4-c0a3df528f00-kube-api-access-582jc\") pod \"kserve-controller-manager-d6f4b65b8-28rcq\" (UID: \"22f96003-9eb5-47dd-a1c4-c0a3df528f00\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:16:59.484047 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.484020 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-59dph"] Jul 14 22:16:59.484219 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.484195 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" podUID="42984c53-43dc-4c20-bfa8-e3382be041c5" containerName="discovery" containerID="cri-o://4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39" gracePeriod=30 Jul 14 22:16:59.512619 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.512590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:16:59.691132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.690972 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-28rcq"] Jul 14 22:16:59.693845 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:16:59.693816 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f96003_9eb5_47dd_a1c4_c0a3df528f00.slice/crio-6638e46741c4c56f050b8c519c91e56a5eeb53f0b742a96f43538a014b015fa1 WatchSource:0}: Error finding container 6638e46741c4c56f050b8c519c91e56a5eeb53f0b742a96f43538a014b015fa1: Status 404 returned error can't find the container with id 6638e46741c4c56f050b8c519c91e56a5eeb53f0b742a96f43538a014b015fa1 Jul 14 22:16:59.735606 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.735581 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:59.801934 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.801902 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-dns-cert\") pod \"42984c53-43dc-4c20-bfa8-e3382be041c5\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " Jul 14 22:16:59.802089 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.801972 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42984c53-43dc-4c20-bfa8-e3382be041c5-local-certs\") pod \"42984c53-43dc-4c20-bfa8-e3382be041c5\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " Jul 14 22:16:59.802089 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.802007 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-ca-configmap\") pod \"42984c53-43dc-4c20-bfa8-e3382be041c5\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " Jul 14 22:16:59.802089 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.802031 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-kubeconfig\") pod \"42984c53-43dc-4c20-bfa8-e3382be041c5\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " Jul 14 22:16:59.802089 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.802059 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-token\") pod \"42984c53-43dc-4c20-bfa8-e3382be041c5\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " Jul 14 22:16:59.802305 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.802106 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-cacerts\") pod \"42984c53-43dc-4c20-bfa8-e3382be041c5\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " Jul 14 22:16:59.802305 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.802133 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w524m\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-kube-api-access-w524m\") pod \"42984c53-43dc-4c20-bfa8-e3382be041c5\" (UID: \"42984c53-43dc-4c20-bfa8-e3382be041c5\") " Jul 14 22:16:59.802504 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.802465 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "42984c53-43dc-4c20-bfa8-e3382be041c5" (UID: "42984c53-43dc-4c20-bfa8-e3382be041c5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:16:59.804533 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.804505 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42984c53-43dc-4c20-bfa8-e3382be041c5-local-certs" (OuterVolumeSpecName: "local-certs") pod "42984c53-43dc-4c20-bfa8-e3382be041c5" (UID: "42984c53-43dc-4c20-bfa8-e3382be041c5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:16:59.804636 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.804582 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "42984c53-43dc-4c20-bfa8-e3382be041c5" (UID: "42984c53-43dc-4c20-bfa8-e3382be041c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:16:59.804710 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.804690 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "42984c53-43dc-4c20-bfa8-e3382be041c5" (UID: "42984c53-43dc-4c20-bfa8-e3382be041c5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:16:59.804798 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.804754 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "42984c53-43dc-4c20-bfa8-e3382be041c5" (UID: "42984c53-43dc-4c20-bfa8-e3382be041c5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:16:59.804798 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.804782 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-cacerts" (OuterVolumeSpecName: "cacerts") pod "42984c53-43dc-4c20-bfa8-e3382be041c5" (UID: "42984c53-43dc-4c20-bfa8-e3382be041c5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:16:59.804890 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.804856 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-kube-api-access-w524m" (OuterVolumeSpecName: "kube-api-access-w524m") pod "42984c53-43dc-4c20-bfa8-e3382be041c5" (UID: "42984c53-43dc-4c20-bfa8-e3382be041c5"). InnerVolumeSpecName "kube-api-access-w524m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:16:59.884125 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.884089 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" event={"ID":"22f96003-9eb5-47dd-a1c4-c0a3df528f00","Type":"ContainerStarted","Data":"6638e46741c4c56f050b8c519c91e56a5eeb53f0b742a96f43538a014b015fa1"} Jul 14 22:16:59.885709 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.885671 2578 generic.go:356] "Generic (PLEG): container finished" podID="42984c53-43dc-4c20-bfa8-e3382be041c5" containerID="4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39" exitCode=0 Jul 14 22:16:59.885849 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.885736 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" Jul 14 22:16:59.885849 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.885754 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" event={"ID":"42984c53-43dc-4c20-bfa8-e3382be041c5","Type":"ContainerDied","Data":"4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39"} Jul 14 22:16:59.885849 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.885798 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-59dph" event={"ID":"42984c53-43dc-4c20-bfa8-e3382be041c5","Type":"ContainerDied","Data":"23850b42d81fd2e3e174fb16f8bf979f756470e9f6c551b8313d4046b4b31ebc"} Jul 14 22:16:59.885849 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.885821 2578 scope.go:117] "RemoveContainer" containerID="4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39" Jul 14 22:16:59.886683 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.886121 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:59.887517 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.887477 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42sb8v" Jul 14 22:16:59.895211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.895193 2578 scope.go:117] "RemoveContainer" containerID="4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39" Jul 14 22:16:59.895477 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:16:59.895459 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39\": container with ID starting with 4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39 not found: ID does not exist" containerID="4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39" Jul 14 22:16:59.895541 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.895488 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39"} err="failed to get container status \"4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39\": rpc error: code = NotFound desc = could not find container \"4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39\": container with ID starting with 4d1fab0b9af447b0eecd259d5a36e6cc8fd4847eb6035637164ac876a15cce39 not found: ID does not exist" Jul 14 22:16:59.903848 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.903824 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-dns-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:16:59.903848 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.903849 2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42984c53-43dc-4c20-bfa8-e3382be041c5-local-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:16:59.903989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.903864 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-csr-ca-configmap\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:16:59.903989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.903876 2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-kubeconfig\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:16:59.903989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.903890 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:16:59.903989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.903947 2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42984c53-43dc-4c20-bfa8-e3382be041c5-cacerts\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:16:59.903989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.903960 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w524m\" (UniqueName: \"kubernetes.io/projected/42984c53-43dc-4c20-bfa8-e3382be041c5-kube-api-access-w524m\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:16:59.959312 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.959276 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-59dph"] Jul 14 22:16:59.969961 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:16:59.969926 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-59dph"] Jul 14 22:17:00.838379 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:00.838344 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42984c53-43dc-4c20-bfa8-e3382be041c5" path="/var/lib/kubelet/pods/42984c53-43dc-4c20-bfa8-e3382be041c5/volumes" Jul 14 22:17:02.847062 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:02.847032 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6pwzw" Jul 14 22:17:02.911598 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:02.911558 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" event={"ID":"22f96003-9eb5-47dd-a1c4-c0a3df528f00","Type":"ContainerStarted","Data":"5d53a19fd038344f078bda95f580bfd7d5e3d8f9fba8e73678794c9680e94520"} Jul 14 22:17:02.911801 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:02.911699 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:17:02.963215 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:02.963165 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" podStartSLOduration=1.357673813 podStartE2EDuration="3.963148366s" totalImagesPullingTime="2.605474553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:17:02.962015379 +0000 UTC m=+488.774647018" watchObservedRunningTime="2026-07-14 22:17:02.963148366 +0000 UTC m=+488.775780005" Jul 14 22:17:33.921968 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:33.921931 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-28rcq" Jul 14 22:17:47.385094 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.385016 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s989r"] Jul 14 22:17:47.385533 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.385374 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42984c53-43dc-4c20-bfa8-e3382be041c5" containerName="discovery" Jul 14 22:17:47.385533 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.385386 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="42984c53-43dc-4c20-bfa8-e3382be041c5" containerName="discovery" Jul 14 22:17:47.385533 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.385448 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="42984c53-43dc-4c20-bfa8-e3382be041c5" containerName="discovery" Jul 14 22:17:47.398861 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.398835 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s989r" Jul 14 22:17:47.413709 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.413678 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-nbcrc\"" Jul 14 22:17:47.414528 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.414506 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s989r"] Jul 14 22:17:47.414878 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.414855 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 22:17:47.414986 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.414920 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 22:17:47.498838 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.498800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z5qn\" (UniqueName: \"kubernetes.io/projected/2d9772d2-dae3-41f6-b6cf-cb281d2b41af-kube-api-access-2z5qn\") pod \"kuadrant-operator-catalog-s989r\" (UID: \"2d9772d2-dae3-41f6-b6cf-cb281d2b41af\") " pod="kuadrant-system/kuadrant-operator-catalog-s989r" Jul 14 22:17:47.599317 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.599282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z5qn\" (UniqueName: \"kubernetes.io/projected/2d9772d2-dae3-41f6-b6cf-cb281d2b41af-kube-api-access-2z5qn\") pod \"kuadrant-operator-catalog-s989r\" (UID: \"2d9772d2-dae3-41f6-b6cf-cb281d2b41af\") " pod="kuadrant-system/kuadrant-operator-catalog-s989r" Jul 14 22:17:47.615562 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.615530 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z5qn\" (UniqueName: \"kubernetes.io/projected/2d9772d2-dae3-41f6-b6cf-cb281d2b41af-kube-api-access-2z5qn\") pod \"kuadrant-operator-catalog-s989r\" (UID: \"2d9772d2-dae3-41f6-b6cf-cb281d2b41af\") " pod="kuadrant-system/kuadrant-operator-catalog-s989r" Jul 14 22:17:47.664195 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.664123 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s989r"] Jul 14 22:17:47.664386 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.664372 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s989r" Jul 14 22:17:47.798104 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.798060 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s989r"] Jul 14 22:17:47.800734 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:17:47.800700 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9772d2_dae3_41f6_b6cf_cb281d2b41af.slice/crio-11c0416529540ff1a77e8f549ef4b1c73794dffd7002642609c939e8236948d5 WatchSource:0}: Error finding container 11c0416529540ff1a77e8f549ef4b1c73794dffd7002642609c939e8236948d5: Status 404 returned error can't find the container with id 11c0416529540ff1a77e8f549ef4b1c73794dffd7002642609c939e8236948d5 Jul 14 22:17:47.875522 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.875488 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gzwpr"] Jul 14 22:17:47.880280 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.880262 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" Jul 14 22:17:47.895330 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:47.895307 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gzwpr"] Jul 14 22:17:48.002792 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:48.002717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kc9j\" (UniqueName: \"kubernetes.io/projected/60f91db4-18c6-4850-9d1c-cd013558f674-kube-api-access-4kc9j\") pod \"kuadrant-operator-catalog-gzwpr\" (UID: \"60f91db4-18c6-4850-9d1c-cd013558f674\") " pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" Jul 14 22:17:48.103685 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:48.103650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc9j\" (UniqueName: \"kubernetes.io/projected/60f91db4-18c6-4850-9d1c-cd013558f674-kube-api-access-4kc9j\") pod \"kuadrant-operator-catalog-gzwpr\" (UID: \"60f91db4-18c6-4850-9d1c-cd013558f674\") " pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" Jul 14 22:17:48.115428 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:48.115388 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s989r" event={"ID":"2d9772d2-dae3-41f6-b6cf-cb281d2b41af","Type":"ContainerStarted","Data":"11c0416529540ff1a77e8f549ef4b1c73794dffd7002642609c939e8236948d5"} Jul 14 22:17:48.116730 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:48.116710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kc9j\" (UniqueName: \"kubernetes.io/projected/60f91db4-18c6-4850-9d1c-cd013558f674-kube-api-access-4kc9j\") pod \"kuadrant-operator-catalog-gzwpr\" (UID: \"60f91db4-18c6-4850-9d1c-cd013558f674\") " pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" Jul 14 22:17:48.189739 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:48.189706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" Jul 14 22:17:48.319606 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:48.319580 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gzwpr"] Jul 14 22:17:48.321371 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:17:48.321344 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f91db4_18c6_4850_9d1c_cd013558f674.slice/crio-7d6376e0c9c39acc91c0a277000f818b7b1c88017c18a9b7d492c8a508b7a36d WatchSource:0}: Error finding container 7d6376e0c9c39acc91c0a277000f818b7b1c88017c18a9b7d492c8a508b7a36d: Status 404 returned error can't find the container with id 7d6376e0c9c39acc91c0a277000f818b7b1c88017c18a9b7d492c8a508b7a36d Jul 14 22:17:49.121201 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:49.121163 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" event={"ID":"60f91db4-18c6-4850-9d1c-cd013558f674","Type":"ContainerStarted","Data":"7d6376e0c9c39acc91c0a277000f818b7b1c88017c18a9b7d492c8a508b7a36d"} Jul 14 22:17:51.130572 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:51.130529 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s989r" event={"ID":"2d9772d2-dae3-41f6-b6cf-cb281d2b41af","Type":"ContainerStarted","Data":"0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448"} Jul 14 22:17:51.131075 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:51.130583 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-s989r" podUID="2d9772d2-dae3-41f6-b6cf-cb281d2b41af" containerName="registry-server" containerID="cri-o://0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448" gracePeriod=2 Jul 14 22:17:51.132031 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:51.132011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" event={"ID":"60f91db4-18c6-4850-9d1c-cd013558f674","Type":"ContainerStarted","Data":"8423a6cb425818cfcca15577b8395f18a2f7817d8d9eb789ba4021794ad0c2f9"} Jul 14 22:17:51.154736 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:51.154695 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-s989r" podStartSLOduration=1.8643704190000001 podStartE2EDuration="4.154681041s" totalImagesPullingTime="2.290310622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:17:51.151706523 +0000 UTC m=+536.964338162" watchObservedRunningTime="2026-07-14 22:17:51.154681041 +0000 UTC m=+536.967312681" Jul 14 22:17:51.178438 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:51.178396 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" podStartSLOduration=2.4051499290000002 podStartE2EDuration="4.178382504s" totalImagesPullingTime="1.773232575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:17:51.177774844 +0000 UTC m=+536.990406483" watchObservedRunningTime="2026-07-14 22:17:51.178382504 +0000 UTC m=+536.991014145" Jul 14 22:17:51.378117 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:51.378091 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s989r" Jul 14 22:17:51.432192 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:51.432120 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z5qn\" (UniqueName: \"kubernetes.io/projected/2d9772d2-dae3-41f6-b6cf-cb281d2b41af-kube-api-access-2z5qn\") pod \"2d9772d2-dae3-41f6-b6cf-cb281d2b41af\" (UID: \"2d9772d2-dae3-41f6-b6cf-cb281d2b41af\") " Jul 14 22:17:51.434231 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:51.434205 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9772d2-dae3-41f6-b6cf-cb281d2b41af-kube-api-access-2z5qn" (OuterVolumeSpecName: "kube-api-access-2z5qn") pod "2d9772d2-dae3-41f6-b6cf-cb281d2b41af" (UID: "2d9772d2-dae3-41f6-b6cf-cb281d2b41af"). InnerVolumeSpecName "kube-api-access-2z5qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:17:51.533101 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:51.533073 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z5qn\" (UniqueName: \"kubernetes.io/projected/2d9772d2-dae3-41f6-b6cf-cb281d2b41af-kube-api-access-2z5qn\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:17:52.137302 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.137269 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d9772d2-dae3-41f6-b6cf-cb281d2b41af" containerID="0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448" exitCode=0 Jul 14 22:17:52.137710 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.137365 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s989r" Jul 14 22:17:52.137710 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.137401 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s989r" event={"ID":"2d9772d2-dae3-41f6-b6cf-cb281d2b41af","Type":"ContainerDied","Data":"0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448"} Jul 14 22:17:52.137710 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.137447 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s989r" event={"ID":"2d9772d2-dae3-41f6-b6cf-cb281d2b41af","Type":"ContainerDied","Data":"11c0416529540ff1a77e8f549ef4b1c73794dffd7002642609c939e8236948d5"} Jul 14 22:17:52.137710 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.137467 2578 scope.go:117] "RemoveContainer" containerID="0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448" Jul 14 22:17:52.146858 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.146838 2578 scope.go:117] "RemoveContainer" containerID="0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448" Jul 14 22:17:52.147104 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:17:52.147084 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448\": container with ID starting with 0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448 not found: ID does not exist" containerID="0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448" Jul 14 22:17:52.147158 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.147117 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448"} err="failed to get container status \"0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448\": rpc error: code = NotFound desc = could not find container \"0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448\": container with ID starting with 0405d8589b0269322fc9e77a2f95d60649027d8951ec125de3670d5afba48448 not found: ID does not exist" Jul 14 22:17:52.159610 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.159578 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s989r"] Jul 14 22:17:52.164535 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.164516 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s989r"] Jul 14 22:17:52.836805 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:52.836772 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9772d2-dae3-41f6-b6cf-cb281d2b41af" path="/var/lib/kubelet/pods/2d9772d2-dae3-41f6-b6cf-cb281d2b41af/volumes" Jul 14 22:17:58.189979 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:58.189944 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" Jul 14 22:17:58.190374 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:58.190000 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" Jul 14 22:17:58.212448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:58.212424 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" Jul 14 22:17:59.189173 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:17:59.189143 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-gzwpr" Jul 14 22:18:19.563970 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.563936 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6"] Jul 14 22:18:19.564438 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.564276 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9772d2-dae3-41f6-b6cf-cb281d2b41af" containerName="registry-server" Jul 14 22:18:19.564438 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.564287 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9772d2-dae3-41f6-b6cf-cb281d2b41af" containerName="registry-server" Jul 14 22:18:19.564438 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.564344 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9772d2-dae3-41f6-b6cf-cb281d2b41af" containerName="registry-server" Jul 14 22:18:19.567132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.567114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:19.570579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.570554 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kshmc\"" Jul 14 22:18:19.582461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.582441 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6"] Jul 14 22:18:19.654867 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.654839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpskv\" (UniqueName: \"kubernetes.io/projected/d734bfbc-b10a-4f38-9f32-fa4d349435aa-kube-api-access-mpskv\") pod \"limitador-operator-controller-manager-859d7b78bd-gzgn6\" (UID: \"d734bfbc-b10a-4f38-9f32-fa4d349435aa\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:19.755868 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.755844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpskv\" (UniqueName: \"kubernetes.io/projected/d734bfbc-b10a-4f38-9f32-fa4d349435aa-kube-api-access-mpskv\") pod \"limitador-operator-controller-manager-859d7b78bd-gzgn6\" (UID: \"d734bfbc-b10a-4f38-9f32-fa4d349435aa\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:19.769584 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.769566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpskv\" (UniqueName: \"kubernetes.io/projected/d734bfbc-b10a-4f38-9f32-fa4d349435aa-kube-api-access-mpskv\") pod \"limitador-operator-controller-manager-859d7b78bd-gzgn6\" (UID: \"d734bfbc-b10a-4f38-9f32-fa4d349435aa\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:19.877038 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:19.876968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:20.013276 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:20.013232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6"] Jul 14 22:18:20.017149 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:18:20.017119 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd734bfbc_b10a_4f38_9f32_fa4d349435aa.slice/crio-361f2f7995bbcc863d254defbbea957729ec041e206c687b88c5e82c24e74854 WatchSource:0}: Error finding container 361f2f7995bbcc863d254defbbea957729ec041e206c687b88c5e82c24e74854: Status 404 returned error can't find the container with id 361f2f7995bbcc863d254defbbea957729ec041e206c687b88c5e82c24e74854 Jul 14 22:18:20.257696 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:20.257601 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" event={"ID":"d734bfbc-b10a-4f38-9f32-fa4d349435aa","Type":"ContainerStarted","Data":"361f2f7995bbcc863d254defbbea957729ec041e206c687b88c5e82c24e74854"} Jul 14 22:18:22.273228 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:22.273119 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" event={"ID":"d734bfbc-b10a-4f38-9f32-fa4d349435aa","Type":"ContainerStarted","Data":"b3053b85c43621d209c22608dddfc47f8b19cf58bde8edf306b5db176f7acbf5"} Jul 14 22:18:22.273697 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:22.273267 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:22.296759 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:22.296719 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" podStartSLOduration=1.313022446 podStartE2EDuration="3.296707036s" totalImagesPullingTime="1.98368459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:22.295730611 +0000 UTC m=+568.108362251" watchObservedRunningTime="2026-07-14 22:18:22.296707036 +0000 UTC m=+568.109338676" Jul 14 22:18:23.132158 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.132126 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f"] Jul 14 22:18:23.135716 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.135692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" Jul 14 22:18:23.139310 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.139285 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-l5825\"" Jul 14 22:18:23.139310 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.139306 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 14 22:18:23.155071 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.155038 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f"] Jul 14 22:18:23.286510 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.286476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh7nk\" (UniqueName: \"kubernetes.io/projected/ac70abc1-baf6-4d3d-8477-abd13a315f2a-kube-api-access-wh7nk\") pod \"dns-operator-controller-manager-864dd78c4c-tbb2f\" (UID: \"ac70abc1-baf6-4d3d-8477-abd13a315f2a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" Jul 14 22:18:23.387222 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.387142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7nk\" (UniqueName: \"kubernetes.io/projected/ac70abc1-baf6-4d3d-8477-abd13a315f2a-kube-api-access-wh7nk\") pod \"dns-operator-controller-manager-864dd78c4c-tbb2f\" (UID: \"ac70abc1-baf6-4d3d-8477-abd13a315f2a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" Jul 14 22:18:23.408664 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.408635 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh7nk\" (UniqueName: \"kubernetes.io/projected/ac70abc1-baf6-4d3d-8477-abd13a315f2a-kube-api-access-wh7nk\") pod \"dns-operator-controller-manager-864dd78c4c-tbb2f\" (UID: \"ac70abc1-baf6-4d3d-8477-abd13a315f2a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" Jul 14 22:18:23.445354 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.445333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" Jul 14 22:18:23.589208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:23.589181 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f"] Jul 14 22:18:23.591463 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:18:23.591440 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac70abc1_baf6_4d3d_8477_abd13a315f2a.slice/crio-2b265ee1b378a668a7b2012c5386eafebbb5b8ad6eb542a5e69c916ddd26ccb5 WatchSource:0}: Error finding container 2b265ee1b378a668a7b2012c5386eafebbb5b8ad6eb542a5e69c916ddd26ccb5: Status 404 returned error can't find the container with id 2b265ee1b378a668a7b2012c5386eafebbb5b8ad6eb542a5e69c916ddd26ccb5 Jul 14 22:18:24.283585 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:24.283552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" event={"ID":"ac70abc1-baf6-4d3d-8477-abd13a315f2a","Type":"ContainerStarted","Data":"2b265ee1b378a668a7b2012c5386eafebbb5b8ad6eb542a5e69c916ddd26ccb5"} Jul 14 22:18:26.293480 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:26.293393 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" event={"ID":"ac70abc1-baf6-4d3d-8477-abd13a315f2a","Type":"ContainerStarted","Data":"27b1ad23b9201af62ffa65e140a4a979f89edd80c4a0e07cf6411abc1d6ea3dd"} Jul 14 22:18:26.293866 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:26.293490 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" Jul 14 22:18:26.327408 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:26.327358 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" podStartSLOduration=1.063267516 podStartE2EDuration="3.327339232s" totalImagesPullingTime="2.264071716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:26.324969395 +0000 UTC m=+572.137601046" watchObservedRunningTime="2026-07-14 22:18:26.327339232 +0000 UTC m=+572.139970875" Jul 14 22:18:26.855305 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:26.855271 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-k9p69"] Jul 14 22:18:26.858654 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:26.858639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" Jul 14 22:18:26.862179 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:26.862163 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-p9ckn\"" Jul 14 22:18:26.879440 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:26.879417 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-k9p69"] Jul 14 22:18:27.020207 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:27.020172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjlk7\" (UniqueName: \"kubernetes.io/projected/0e46c6ea-0de1-48bc-b4e7-89c5901c029d-kube-api-access-jjlk7\") pod \"authorino-operator-75cc8d5cb7-k9p69\" (UID: \"0e46c6ea-0de1-48bc-b4e7-89c5901c029d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" Jul 14 22:18:27.121446 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:27.121361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjlk7\" (UniqueName: \"kubernetes.io/projected/0e46c6ea-0de1-48bc-b4e7-89c5901c029d-kube-api-access-jjlk7\") pod \"authorino-operator-75cc8d5cb7-k9p69\" (UID: \"0e46c6ea-0de1-48bc-b4e7-89c5901c029d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" Jul 14 22:18:27.136664 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:27.136634 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjlk7\" (UniqueName: \"kubernetes.io/projected/0e46c6ea-0de1-48bc-b4e7-89c5901c029d-kube-api-access-jjlk7\") pod \"authorino-operator-75cc8d5cb7-k9p69\" (UID: \"0e46c6ea-0de1-48bc-b4e7-89c5901c029d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" Jul 14 22:18:27.168456 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:27.168435 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" Jul 14 22:18:27.304859 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:27.304834 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-k9p69"] Jul 14 22:18:28.304444 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:28.304401 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" event={"ID":"0e46c6ea-0de1-48bc-b4e7-89c5901c029d","Type":"ContainerStarted","Data":"41191fbec83a15a5bf0f26d341868f3036a8e78d76ff1d92f268a3048b443f1a"} Jul 14 22:18:30.316098 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:30.316064 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" event={"ID":"0e46c6ea-0de1-48bc-b4e7-89c5901c029d","Type":"ContainerStarted","Data":"0c009b0c43c77da5840f71fd89ad9e4da4e5144d1e1cf3952a705099f072d41c"} Jul 14 22:18:30.316502 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:30.316197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" Jul 14 22:18:30.338333 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:30.338286 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" podStartSLOduration=1.861699306 podStartE2EDuration="4.338271818s" totalImagesPullingTime="2.476572512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:30.336669275 +0000 UTC m=+576.149300914" watchObservedRunningTime="2026-07-14 22:18:30.338271818 +0000 UTC m=+576.150903457" Jul 14 22:18:32.488321 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:32.488292 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss"] Jul 14 22:18:32.491731 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:32.491716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:32.495314 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:32.495286 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:32.499722 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:32.499699 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-n4n2r\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4n2r\"" type="*v1.Secret" Jul 14 22:18:32.523118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:32.523096 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss"] Jul 14 22:18:32.671995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:32.671959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8h2\" (UniqueName: \"kubernetes.io/projected/c157d2c7-92f9-4c6b-84c8-5201db1e854c-kube-api-access-9f8h2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" (UID: \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:32.672179 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:32.672041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c157d2c7-92f9-4c6b-84c8-5201db1e854c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" (UID: \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:32.773095 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:32.772997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c157d2c7-92f9-4c6b-84c8-5201db1e854c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" (UID: \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:32.773306 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:32.773144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8h2\" (UniqueName: \"kubernetes.io/projected/c157d2c7-92f9-4c6b-84c8-5201db1e854c-kube-api-access-9f8h2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" (UID: \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:32.773487 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:32.773464 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c157d2c7-92f9-4c6b-84c8-5201db1e854c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" (UID: \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:32.791450 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:32.791420 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8h2\" (UniqueName: \"kubernetes.io/projected/c157d2c7-92f9-4c6b-84c8-5201db1e854c-kube-api-access-9f8h2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" (UID: \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:33.280554 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:33.280519 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:33.803035 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:33.802998 2578 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" secret="" err="failed to sync secret cache: timed out waiting for the condition" Jul 14 22:18:33.803450 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:33.803061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:33.821624 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:33.821600 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4n2r\"" Jul 14 22:18:33.956832 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:33.956810 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss"] Jul 14 22:18:33.958945 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:18:33.958917 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc157d2c7_92f9_4c6b_84c8_5201db1e854c.slice/crio-c152f14500e4548021a85d48dd6f18d29d2dd48708ae2dd7994543073c45b705 WatchSource:0}: Error finding container c152f14500e4548021a85d48dd6f18d29d2dd48708ae2dd7994543073c45b705: Status 404 returned error can't find the container with id c152f14500e4548021a85d48dd6f18d29d2dd48708ae2dd7994543073c45b705 Jul 14 22:18:34.335365 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:34.335332 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" event={"ID":"c157d2c7-92f9-4c6b-84c8-5201db1e854c","Type":"ContainerStarted","Data":"c152f14500e4548021a85d48dd6f18d29d2dd48708ae2dd7994543073c45b705"} Jul 14 22:18:37.300611 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:37.300547 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tbb2f" Jul 14 22:18:41.323341 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:41.323306 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-k9p69" Jul 14 22:18:41.373372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:41.373336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" event={"ID":"c157d2c7-92f9-4c6b-84c8-5201db1e854c","Type":"ContainerStarted","Data":"c9acd7edfff63d01884e8b3cad2c9bd213292d4774bffdf4a3915e48ecacca33"} Jul 14 22:18:41.373542 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:41.373463 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:41.404431 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:41.404384 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" podStartSLOduration=2.815456599 podStartE2EDuration="9.404368226s" totalImagesPullingTime="6.588911627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:41.401819924 +0000 UTC m=+587.214451564" watchObservedRunningTime="2026-07-14 22:18:41.404368226 +0000 UTC m=+587.216999866" Jul 14 22:18:52.380972 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:52.380941 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:54.287732 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.287698 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss"] Jul 14 22:18:54.288099 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.287930 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" containerName="manager" containerID="cri-o://c9acd7edfff63d01884e8b3cad2c9bd213292d4774bffdf4a3915e48ecacca33" gracePeriod=2 Jul 14 22:18:54.293921 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.293895 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss"] Jul 14 22:18:54.311131 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.311107 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6"] Jul 14 22:18:54.311348 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.311323 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" podUID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" containerName="manager" containerID="cri-o://b3053b85c43621d209c22608dddfc47f8b19cf58bde8edf306b5db176f7acbf5" gracePeriod=2 Jul 14 22:18:54.329649 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.329619 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l"] Jul 14 22:18:54.330133 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.330091 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" containerName="manager" Jul 14 22:18:54.330133 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.330112 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" containerName="manager" Jul 14 22:18:54.330325 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.330214 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" containerName="manager" Jul 14 22:18:54.333519 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.333498 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:18:54.334633 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.334587 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6"] Jul 14 22:18:54.341342 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.341312 2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d734bfbc-b10a-4f38-9f32-fa4d349435aa\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-gzgn6\": pods \"limitador-operator-controller-manager-859d7b78bd-gzgn6\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:54.343889 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.343861 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:54.349761 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.349736 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:54.368762 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.368740 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l"] Jul 14 22:18:54.405235 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.405205 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn6\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:54.437734 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.437687 2578 generic.go:356] "Generic (PLEG): container finished" podID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" containerID="b3053b85c43621d209c22608dddfc47f8b19cf58bde8edf306b5db176f7acbf5" exitCode=0 Jul 14 22:18:54.439647 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.439611 2578 generic.go:356] "Generic (PLEG): container finished" podID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" containerID="c9acd7edfff63d01884e8b3cad2c9bd213292d4774bffdf4a3915e48ecacca33" exitCode=0 Jul 14 22:18:54.447144 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.447113 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmczb\" (UniqueName: \"kubernetes.io/projected/1c0be113-8327-462f-9c2a-95dde0e88e79-kube-api-access-fmczb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nhs9l\" (UID: \"1c0be113-8327-462f-9c2a-95dde0e88e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:18:54.447286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.447186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0be113-8327-462f-9c2a-95dde0e88e79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nhs9l\" (UID: \"1c0be113-8327-462f-9c2a-95dde0e88e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:18:54.546510 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.546488 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:54.548031 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.548008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0be113-8327-462f-9c2a-95dde0e88e79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nhs9l\" (UID: \"1c0be113-8327-462f-9c2a-95dde0e88e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:18:54.548121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.548100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmczb\" (UniqueName: \"kubernetes.io/projected/1c0be113-8327-462f-9c2a-95dde0e88e79-kube-api-access-fmczb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nhs9l\" (UID: \"1c0be113-8327-462f-9c2a-95dde0e88e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:18:54.548396 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.548374 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0be113-8327-462f-9c2a-95dde0e88e79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nhs9l\" (UID: \"1c0be113-8327-462f-9c2a-95dde0e88e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:18:54.549383 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.549360 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn6\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:54.551857 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.551824 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:54.559863 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.559843 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmczb\" (UniqueName: \"kubernetes.io/projected/1c0be113-8327-462f-9c2a-95dde0e88e79-kube-api-access-fmczb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nhs9l\" (UID: \"1c0be113-8327-462f-9c2a-95dde0e88e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:18:54.559980 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.559965 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:54.562523 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.562500 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn6\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:54.564879 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.564852 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:54.648406 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.648381 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c157d2c7-92f9-4c6b-84c8-5201db1e854c-extensions-socket-volume\") pod \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\" (UID: \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\") " Jul 14 22:18:54.648561 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.648433 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpskv\" (UniqueName: \"kubernetes.io/projected/d734bfbc-b10a-4f38-9f32-fa4d349435aa-kube-api-access-mpskv\") pod \"d734bfbc-b10a-4f38-9f32-fa4d349435aa\" (UID: \"d734bfbc-b10a-4f38-9f32-fa4d349435aa\") " Jul 14 22:18:54.648561 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.648472 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f8h2\" (UniqueName: \"kubernetes.io/projected/c157d2c7-92f9-4c6b-84c8-5201db1e854c-kube-api-access-9f8h2\") pod \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\" (UID: \"c157d2c7-92f9-4c6b-84c8-5201db1e854c\") " Jul 14 22:18:54.648886 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.648861 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c157d2c7-92f9-4c6b-84c8-5201db1e854c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c157d2c7-92f9-4c6b-84c8-5201db1e854c" (UID: "c157d2c7-92f9-4c6b-84c8-5201db1e854c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:54.650708 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.650682 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d734bfbc-b10a-4f38-9f32-fa4d349435aa-kube-api-access-mpskv" (OuterVolumeSpecName: "kube-api-access-mpskv") pod "d734bfbc-b10a-4f38-9f32-fa4d349435aa" (UID: "d734bfbc-b10a-4f38-9f32-fa4d349435aa"). InnerVolumeSpecName "kube-api-access-mpskv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:18:54.650946 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.650927 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c157d2c7-92f9-4c6b-84c8-5201db1e854c-kube-api-access-9f8h2" (OuterVolumeSpecName: "kube-api-access-9f8h2") pod "c157d2c7-92f9-4c6b-84c8-5201db1e854c" (UID: "c157d2c7-92f9-4c6b-84c8-5201db1e854c"). InnerVolumeSpecName "kube-api-access-9f8h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:18:54.651985 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.651968 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:18:54.652951 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.652934 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:18:54.718530 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.718505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:18:54.749868 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.749844 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c157d2c7-92f9-4c6b-84c8-5201db1e854c-extensions-socket-volume\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:18:54.749868 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.749867 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpskv\" (UniqueName: \"kubernetes.io/projected/d734bfbc-b10a-4f38-9f32-fa4d349435aa-kube-api-access-mpskv\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:18:54.750048 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.749877 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8h2\" (UniqueName: \"kubernetes.io/projected/c157d2c7-92f9-4c6b-84c8-5201db1e854c-kube-api-access-9f8h2\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:18:54.837449 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.837416 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" path="/var/lib/kubelet/pods/c157d2c7-92f9-4c6b-84c8-5201db1e854c/volumes" Jul 14 22:18:54.837828 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.837803 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" path="/var/lib/kubelet/pods/d734bfbc-b10a-4f38-9f32-fa4d349435aa/volumes" Jul 14 22:18:54.839948 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.839914 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn6\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:54.842235 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:18:54.842209 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8t4ss\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="c157d2c7-92f9-4c6b-84c8-5201db1e854c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:54.867648 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:54.867622 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l"] Jul 14 22:18:54.889829 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:18:54.889805 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0be113_8327_462f_9c2a_95dde0e88e79.slice/crio-8a6a70390d7a83c072923282cadb77bb00a513b9f6a628afde9ade67c0cbe558 WatchSource:0}: Error finding container 8a6a70390d7a83c072923282cadb77bb00a513b9f6a628afde9ade67c0cbe558: Status 404 returned error can't find the container with id 8a6a70390d7a83c072923282cadb77bb00a513b9f6a628afde9ade67c0cbe558 Jul 14 22:18:55.445998 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:55.445960 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8t4ss" Jul 14 22:18:55.446403 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:55.445962 2578 scope.go:117] "RemoveContainer" containerID="c9acd7edfff63d01884e8b3cad2c9bd213292d4774bffdf4a3915e48ecacca33" Jul 14 22:18:55.448204 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:55.447918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" event={"ID":"1c0be113-8327-462f-9c2a-95dde0e88e79","Type":"ContainerStarted","Data":"4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad"} Jul 14 22:18:55.448204 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:55.447973 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" event={"ID":"1c0be113-8327-462f-9c2a-95dde0e88e79","Type":"ContainerStarted","Data":"8a6a70390d7a83c072923282cadb77bb00a513b9f6a628afde9ade67c0cbe558"} Jul 14 22:18:55.448204 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:55.448036 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:18:55.449328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:55.449312 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn6" Jul 14 22:18:55.457331 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:55.457311 2578 scope.go:117] "RemoveContainer" containerID="b3053b85c43621d209c22608dddfc47f8b19cf58bde8edf306b5db176f7acbf5" Jul 14 22:18:55.476550 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:18:55.476508 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" podStartSLOduration=1.47649499 podStartE2EDuration="1.47649499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:55.47560768 +0000 UTC m=+601.288239320" watchObservedRunningTime="2026-07-14 22:18:55.47649499 +0000 UTC m=+601.289126631" Jul 14 22:19:06.458715 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:06.458684 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:19:24.266400 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.266321 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l"] Jul 14 22:19:24.266838 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.266515 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" podUID="1c0be113-8327-462f-9c2a-95dde0e88e79" containerName="manager" containerID="cri-o://4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad" gracePeriod=10 Jul 14 22:19:24.513595 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.513571 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:19:24.584803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.584768 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6"] Jul 14 22:19:24.585287 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.585267 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0be113-8327-462f-9c2a-95dde0e88e79" containerName="manager" Jul 14 22:19:24.585381 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.585289 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0be113-8327-462f-9c2a-95dde0e88e79" containerName="manager" Jul 14 22:19:24.585381 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.585351 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" containerName="manager" Jul 14 22:19:24.585381 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.585361 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" containerName="manager" Jul 14 22:19:24.585494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.585456 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0be113-8327-462f-9c2a-95dde0e88e79" containerName="manager" Jul 14 22:19:24.585494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.585479 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d734bfbc-b10a-4f38-9f32-fa4d349435aa" containerName="manager" Jul 14 22:19:24.588617 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.588601 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:24.590952 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.590931 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c0be113-8327-462f-9c2a-95dde0e88e79" containerID="4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad" exitCode=0 Jul 14 22:19:24.591034 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.590959 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" event={"ID":"1c0be113-8327-462f-9c2a-95dde0e88e79","Type":"ContainerDied","Data":"4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad"} Jul 14 22:19:24.591034 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.590977 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" Jul 14 22:19:24.591034 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.590989 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l" event={"ID":"1c0be113-8327-462f-9c2a-95dde0e88e79","Type":"ContainerDied","Data":"8a6a70390d7a83c072923282cadb77bb00a513b9f6a628afde9ade67c0cbe558"} Jul 14 22:19:24.591034 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.591010 2578 scope.go:117] "RemoveContainer" containerID="4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad" Jul 14 22:19:24.598573 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.598551 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmczb\" (UniqueName: \"kubernetes.io/projected/1c0be113-8327-462f-9c2a-95dde0e88e79-kube-api-access-fmczb\") pod \"1c0be113-8327-462f-9c2a-95dde0e88e79\" (UID: \"1c0be113-8327-462f-9c2a-95dde0e88e79\") " Jul 14 22:19:24.598676 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.598661 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0be113-8327-462f-9c2a-95dde0e88e79-extensions-socket-volume\") pod \"1c0be113-8327-462f-9c2a-95dde0e88e79\" (UID: \"1c0be113-8327-462f-9c2a-95dde0e88e79\") " Jul 14 22:19:24.599001 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.598979 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0be113-8327-462f-9c2a-95dde0e88e79-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1c0be113-8327-462f-9c2a-95dde0e88e79" (UID: "1c0be113-8327-462f-9c2a-95dde0e88e79"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:19:24.600230 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.600211 2578 scope.go:117] "RemoveContainer" containerID="4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad" Jul 14 22:19:24.600524 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:19:24.600500 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad\": container with ID starting with 4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad not found: ID does not exist" containerID="4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad" Jul 14 22:19:24.600588 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.600540 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad"} err="failed to get container status \"4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad\": rpc error: code = NotFound desc = could not find container \"4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad\": container with ID starting with 4dac0990058a7320ea3f87a869705475341ecd98fa68721faff6ddc3d4292fad not found: ID does not exist" Jul 14 22:19:24.600588 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.600505 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0be113-8327-462f-9c2a-95dde0e88e79-kube-api-access-fmczb" (OuterVolumeSpecName: "kube-api-access-fmczb") pod "1c0be113-8327-462f-9c2a-95dde0e88e79" (UID: "1c0be113-8327-462f-9c2a-95dde0e88e79"). InnerVolumeSpecName "kube-api-access-fmczb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:19:24.616429 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.616402 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6"] Jul 14 22:19:24.700197 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.700167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69988\" (UniqueName: \"kubernetes.io/projected/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-kube-api-access-69988\") pod \"kuadrant-operator-controller-manager-86878c769f-8rtl6\" (UID: \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:24.700352 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.700217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8rtl6\" (UID: \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:24.700352 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.700338 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmczb\" (UniqueName: \"kubernetes.io/projected/1c0be113-8327-462f-9c2a-95dde0e88e79-kube-api-access-fmczb\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:19:24.700425 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.700361 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c0be113-8327-462f-9c2a-95dde0e88e79-extensions-socket-volume\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:19:24.801278 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.801217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69988\" (UniqueName: \"kubernetes.io/projected/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-kube-api-access-69988\") pod \"kuadrant-operator-controller-manager-86878c769f-8rtl6\" (UID: \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:24.801418 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.801321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8rtl6\" (UID: \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:24.801723 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.801693 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8rtl6\" (UID: \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:24.811825 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.811801 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69988\" (UniqueName: \"kubernetes.io/projected/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-kube-api-access-69988\") pod \"kuadrant-operator-controller-manager-86878c769f-8rtl6\" (UID: \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:24.899678 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.899656 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:24.909026 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.909003 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l"] Jul 14 22:19:24.918601 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:24.918579 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nhs9l"] Jul 14 22:19:25.036707 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:25.036683 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6"] Jul 14 22:19:25.038946 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:19:25.038914 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea6547b_2bf0_49c1_9664_ac4d8cc525aa.slice/crio-9da9c9e6579817bec3c286b23afb7222df34a37e32c31ba117a51d7ec7e057b0 WatchSource:0}: Error finding container 9da9c9e6579817bec3c286b23afb7222df34a37e32c31ba117a51d7ec7e057b0: Status 404 returned error can't find the container with id 9da9c9e6579817bec3c286b23afb7222df34a37e32c31ba117a51d7ec7e057b0 Jul 14 22:19:25.598769 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:25.598731 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" event={"ID":"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa","Type":"ContainerStarted","Data":"3eafbcd5139f715705acb4f3915fadc0010b0257a5a4a48d908cb7363b6bc6ac"} Jul 14 22:19:25.599142 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:25.598791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" event={"ID":"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa","Type":"ContainerStarted","Data":"9da9c9e6579817bec3c286b23afb7222df34a37e32c31ba117a51d7ec7e057b0"} Jul 14 22:19:25.599142 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:25.598820 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:25.627492 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:25.627446 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" podStartSLOduration=1.627431265 podStartE2EDuration="1.627431265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:19:25.626024204 +0000 UTC m=+631.438655882" watchObservedRunningTime="2026-07-14 22:19:25.627431265 +0000 UTC m=+631.440062905" Jul 14 22:19:26.836990 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:26.836954 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c0be113-8327-462f-9c2a-95dde0e88e79" path="/var/lib/kubelet/pods/1c0be113-8327-462f-9c2a-95dde0e88e79/volumes" Jul 14 22:19:36.606345 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:36.606311 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:19:40.771026 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.770989 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf"] Jul 14 22:19:40.774232 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.774216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.777880 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.777855 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ms96v\"" Jul 14 22:19:40.788360 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.788338 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf"] Jul 14 22:19:40.827735 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.827712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.827848 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.827740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.827848 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.827771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.827923 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.827880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.827957 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.827919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.827957 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.827945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.828020 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.827961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wztlp\" (UniqueName: \"kubernetes.io/projected/85c6620a-8dd6-45f2-9862-a34d4e4afe86-kube-api-access-wztlp\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.828020 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.827984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.828085 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.828030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928415 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928697 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928806 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928806 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wztlp\" (UniqueName: \"kubernetes.io/projected/85c6620a-8dd6-45f2-9862-a34d4e4afe86-kube-api-access-wztlp\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928906 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.928906 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.928893 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.929064 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.929031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.929133 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.929078 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.929232 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.929215 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.929497 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.929474 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.931202 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.931184 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.931443 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.931423 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.940714 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.940685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wztlp\" (UniqueName: \"kubernetes.io/projected/85c6620a-8dd6-45f2-9862-a34d4e4afe86-kube-api-access-wztlp\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:40.940792 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:40.940780 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c6620a-8dd6-45f2-9862-a34d4e4afe86-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rjcqf\" (UID: \"85c6620a-8dd6-45f2-9862-a34d4e4afe86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:41.084602 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:41.084576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:41.250611 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:41.250421 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf"] Jul 14 22:19:41.253419 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:19:41.253395 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c6620a_8dd6_45f2_9862_a34d4e4afe86.slice/crio-3e558ec3f9e39ed14477d08423a6dc10f3b234787caf07890781a4279d0e068b WatchSource:0}: Error finding container 3e558ec3f9e39ed14477d08423a6dc10f3b234787caf07890781a4279d0e068b: Status 404 returned error can't find the container with id 3e558ec3f9e39ed14477d08423a6dc10f3b234787caf07890781a4279d0e068b Jul 14 22:19:41.255491 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:41.255459 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:19:41.255555 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:41.255524 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:19:41.255592 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:41.255553 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:19:41.680517 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:41.680478 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" event={"ID":"85c6620a-8dd6-45f2-9862-a34d4e4afe86","Type":"ContainerStarted","Data":"16da43cfb8fce92bf39cca0efc78d2f7136a02363cdd9b0a642342cabc7bb448"} Jul 14 22:19:41.680720 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:41.680530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" event={"ID":"85c6620a-8dd6-45f2-9862-a34d4e4afe86","Type":"ContainerStarted","Data":"3e558ec3f9e39ed14477d08423a6dc10f3b234787caf07890781a4279d0e068b"} Jul 14 22:19:41.707205 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:41.707154 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" podStartSLOduration=1.707138276 podStartE2EDuration="1.707138276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:19:41.703931536 +0000 UTC m=+647.516563179" watchObservedRunningTime="2026-07-14 22:19:41.707138276 +0000 UTC m=+647.519769915" Jul 14 22:19:42.085417 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:42.085390 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:42.090336 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:42.090313 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:42.685471 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:42.685444 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:42.686479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:42.686457 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:19:44.735023 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.734994 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:19:44.738577 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.738556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:19:44.742796 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.742771 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mvnzc\"" Jul 14 22:19:44.742988 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.742968 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 22:19:44.753969 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.753945 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:19:44.761514 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.761489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9442x\" (UniqueName: \"kubernetes.io/projected/2da0ff5f-9d35-485b-a678-56f73d62ef83-kube-api-access-9442x\") pod \"limitador-limitador-54d6d988d5-nxvr2\" (UID: \"2da0ff5f-9d35-485b-a678-56f73d62ef83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:19:44.761695 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.761668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2da0ff5f-9d35-485b-a678-56f73d62ef83-config-file\") pod \"limitador-limitador-54d6d988d5-nxvr2\" (UID: \"2da0ff5f-9d35-485b-a678-56f73d62ef83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:19:44.796729 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.796697 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:19:44.862353 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.862328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9442x\" (UniqueName: \"kubernetes.io/projected/2da0ff5f-9d35-485b-a678-56f73d62ef83-kube-api-access-9442x\") pod \"limitador-limitador-54d6d988d5-nxvr2\" (UID: \"2da0ff5f-9d35-485b-a678-56f73d62ef83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:19:44.862509 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.862490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2da0ff5f-9d35-485b-a678-56f73d62ef83-config-file\") pod \"limitador-limitador-54d6d988d5-nxvr2\" (UID: \"2da0ff5f-9d35-485b-a678-56f73d62ef83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:19:44.863158 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.863142 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2da0ff5f-9d35-485b-a678-56f73d62ef83-config-file\") pod \"limitador-limitador-54d6d988d5-nxvr2\" (UID: \"2da0ff5f-9d35-485b-a678-56f73d62ef83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:19:44.871470 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:44.871450 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9442x\" (UniqueName: \"kubernetes.io/projected/2da0ff5f-9d35-485b-a678-56f73d62ef83-kube-api-access-9442x\") pod \"limitador-limitador-54d6d988d5-nxvr2\" (UID: \"2da0ff5f-9d35-485b-a678-56f73d62ef83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:19:45.048838 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.048753 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:19:45.192079 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.192038 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:19:45.194444 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:19:45.194408 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da0ff5f_9d35_485b_a678_56f73d62ef83.slice/crio-a2a0a6e53f49878e3711d25c85c66ff29338c6498aa149f635b69137e6e0df98 WatchSource:0}: Error finding container a2a0a6e53f49878e3711d25c85c66ff29338c6498aa149f635b69137e6e0df98: Status 404 returned error can't find the container with id a2a0a6e53f49878e3711d25c85c66ff29338c6498aa149f635b69137e6e0df98 Jul 14 22:19:45.273395 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.273363 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-r59r8"] Jul 14 22:19:45.277890 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.277870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r59r8" Jul 14 22:19:45.286188 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.286166 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zjcs2\"" Jul 14 22:19:45.292359 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.292338 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r59r8"] Jul 14 22:19:45.367161 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.367131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xgl\" (UniqueName: \"kubernetes.io/projected/d09cf311-d5c1-4a49-9e26-8daed7831644-kube-api-access-l6xgl\") pod \"authorino-75d556bd77-r59r8\" (UID: \"d09cf311-d5c1-4a49-9e26-8daed7831644\") " pod="kuadrant-system/authorino-75d556bd77-r59r8" Jul 14 22:19:45.468209 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.468181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xgl\" (UniqueName: \"kubernetes.io/projected/d09cf311-d5c1-4a49-9e26-8daed7831644-kube-api-access-l6xgl\") pod \"authorino-75d556bd77-r59r8\" (UID: \"d09cf311-d5c1-4a49-9e26-8daed7831644\") " pod="kuadrant-system/authorino-75d556bd77-r59r8" Jul 14 22:19:45.481140 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.481105 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xgl\" (UniqueName: \"kubernetes.io/projected/d09cf311-d5c1-4a49-9e26-8daed7831644-kube-api-access-l6xgl\") pod \"authorino-75d556bd77-r59r8\" (UID: \"d09cf311-d5c1-4a49-9e26-8daed7831644\") " pod="kuadrant-system/authorino-75d556bd77-r59r8" Jul 14 22:19:45.586739 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.586710 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r59r8" Jul 14 22:19:45.702557 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.702527 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" event={"ID":"2da0ff5f-9d35-485b-a678-56f73d62ef83","Type":"ContainerStarted","Data":"a2a0a6e53f49878e3711d25c85c66ff29338c6498aa149f635b69137e6e0df98"} Jul 14 22:19:45.726113 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:45.726081 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r59r8"] Jul 14 22:19:45.738953 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:19:45.738695 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd09cf311_d5c1_4a49_9e26_8daed7831644.slice/crio-b03aafa8c447ede9c37bad030162bcd062149479e0255b01be5634a84b0cc260 WatchSource:0}: Error finding container b03aafa8c447ede9c37bad030162bcd062149479e0255b01be5634a84b0cc260: Status 404 returned error can't find the container with id b03aafa8c447ede9c37bad030162bcd062149479e0255b01be5634a84b0cc260 Jul 14 22:19:46.712639 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:46.712597 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r59r8" event={"ID":"d09cf311-d5c1-4a49-9e26-8daed7831644","Type":"ContainerStarted","Data":"b03aafa8c447ede9c37bad030162bcd062149479e0255b01be5634a84b0cc260"} Jul 14 22:19:49.730480 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:49.730445 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" event={"ID":"2da0ff5f-9d35-485b-a678-56f73d62ef83","Type":"ContainerStarted","Data":"6a65bddc4ec0909bf0d3f18ca157e86bcd085ce167b70db5f983e1d8531f883a"} Jul 14 22:19:49.730889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:49.730633 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:19:49.731815 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:49.731791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r59r8" event={"ID":"d09cf311-d5c1-4a49-9e26-8daed7831644","Type":"ContainerStarted","Data":"f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a"} Jul 14 22:19:49.754586 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:49.754537 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" podStartSLOduration=1.339715727 podStartE2EDuration="5.75452208s" totalImagesPullingTime="4.414806353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:19:49.754210461 +0000 UTC m=+655.566842102" watchObservedRunningTime="2026-07-14 22:19:49.75452208 +0000 UTC m=+655.567153723" Jul 14 22:19:49.781254 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:19:49.781185 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-r59r8" podStartSLOduration=0.910492546 podStartE2EDuration="4.781170342s" totalImagesPullingTime="3.870677796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:19:49.779969478 +0000 UTC m=+655.592601118" watchObservedRunningTime="2026-07-14 22:19:49.781170342 +0000 UTC m=+655.593801981" Jul 14 22:20:00.737558 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:00.737522 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nxvr2" Jul 14 22:20:17.887029 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:17.886993 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85b78c57d9-rq4fm"] Jul 14 22:20:17.896487 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:17.896464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:20:17.899235 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:17.899215 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 22:20:17.902747 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:17.902724 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85b78c57d9-rq4fm"] Jul 14 22:20:18.043819 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:18.043789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/255e7b01-494b-46c0-b862-61369368a576-tls-cert\") pod \"authorino-85b78c57d9-rq4fm\" (UID: \"255e7b01-494b-46c0-b862-61369368a576\") " pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:20:18.043982 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:18.043846 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mds\" (UniqueName: \"kubernetes.io/projected/255e7b01-494b-46c0-b862-61369368a576-kube-api-access-q8mds\") pod \"authorino-85b78c57d9-rq4fm\" (UID: \"255e7b01-494b-46c0-b862-61369368a576\") " pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:20:18.144861 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:18.144772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/255e7b01-494b-46c0-b862-61369368a576-tls-cert\") pod \"authorino-85b78c57d9-rq4fm\" (UID: \"255e7b01-494b-46c0-b862-61369368a576\") " pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:20:18.144861 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:18.144839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mds\" (UniqueName: \"kubernetes.io/projected/255e7b01-494b-46c0-b862-61369368a576-kube-api-access-q8mds\") pod \"authorino-85b78c57d9-rq4fm\" (UID: \"255e7b01-494b-46c0-b862-61369368a576\") " pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:20:18.147158 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:18.147133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/255e7b01-494b-46c0-b862-61369368a576-tls-cert\") pod \"authorino-85b78c57d9-rq4fm\" (UID: \"255e7b01-494b-46c0-b862-61369368a576\") " pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:20:18.156372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:18.156349 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mds\" (UniqueName: \"kubernetes.io/projected/255e7b01-494b-46c0-b862-61369368a576-kube-api-access-q8mds\") pod \"authorino-85b78c57d9-rq4fm\" (UID: \"255e7b01-494b-46c0-b862-61369368a576\") " pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:20:18.206835 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:18.206802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:20:18.355851 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:18.355816 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85b78c57d9-rq4fm"] Jul 14 22:20:18.358873 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:20:18.358831 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod255e7b01_494b_46c0_b862_61369368a576.slice/crio-3a61445a583f214656576b56cfc078dd62edc9fc0ee7691fcd79443141edb34f WatchSource:0}: Error finding container 3a61445a583f214656576b56cfc078dd62edc9fc0ee7691fcd79443141edb34f: Status 404 returned error can't find the container with id 3a61445a583f214656576b56cfc078dd62edc9fc0ee7691fcd79443141edb34f Jul 14 22:20:18.868019 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:18.867994 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" event={"ID":"255e7b01-494b-46c0-b862-61369368a576","Type":"ContainerStarted","Data":"3a61445a583f214656576b56cfc078dd62edc9fc0ee7691fcd79443141edb34f"} Jul 14 22:20:19.873584 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:19.873550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" event={"ID":"255e7b01-494b-46c0-b862-61369368a576","Type":"ContainerStarted","Data":"f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582"} Jul 14 22:20:19.903469 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:19.903425 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" podStartSLOduration=2.426273618 podStartE2EDuration="2.903408746s" totalImagesPullingTime="477.135128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:20:19.901999112 +0000 UTC m=+685.714630753" watchObservedRunningTime="2026-07-14 22:20:19.903408746 +0000 UTC m=+685.716040387" Jul 14 22:20:19.942597 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:19.942562 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r59r8"] Jul 14 22:20:19.942763 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:19.942742 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-r59r8" podUID="d09cf311-d5c1-4a49-9e26-8daed7831644" containerName="authorino" containerID="cri-o://f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a" gracePeriod=30 Jul 14 22:20:20.194943 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.194917 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r59r8" Jul 14 22:20:20.262773 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.262740 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6xgl\" (UniqueName: \"kubernetes.io/projected/d09cf311-d5c1-4a49-9e26-8daed7831644-kube-api-access-l6xgl\") pod \"d09cf311-d5c1-4a49-9e26-8daed7831644\" (UID: \"d09cf311-d5c1-4a49-9e26-8daed7831644\") " Jul 14 22:20:20.264769 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.264741 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09cf311-d5c1-4a49-9e26-8daed7831644-kube-api-access-l6xgl" (OuterVolumeSpecName: "kube-api-access-l6xgl") pod "d09cf311-d5c1-4a49-9e26-8daed7831644" (UID: "d09cf311-d5c1-4a49-9e26-8daed7831644"). InnerVolumeSpecName "kube-api-access-l6xgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:20:20.363358 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.363324 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6xgl\" (UniqueName: \"kubernetes.io/projected/d09cf311-d5c1-4a49-9e26-8daed7831644-kube-api-access-l6xgl\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:20:20.879556 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.879523 2578 generic.go:356] "Generic (PLEG): container finished" podID="d09cf311-d5c1-4a49-9e26-8daed7831644" containerID="f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a" exitCode=0 Jul 14 22:20:20.879978 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.879565 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r59r8" Jul 14 22:20:20.879978 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.879606 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r59r8" event={"ID":"d09cf311-d5c1-4a49-9e26-8daed7831644","Type":"ContainerDied","Data":"f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a"} Jul 14 22:20:20.879978 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.879647 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r59r8" event={"ID":"d09cf311-d5c1-4a49-9e26-8daed7831644","Type":"ContainerDied","Data":"b03aafa8c447ede9c37bad030162bcd062149479e0255b01be5634a84b0cc260"} Jul 14 22:20:20.879978 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.879665 2578 scope.go:117] "RemoveContainer" containerID="f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a" Jul 14 22:20:20.887771 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.887753 2578 scope.go:117] "RemoveContainer" containerID="f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a" Jul 14 22:20:20.888010 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:20:20.887995 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a\": container with ID starting with f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a not found: ID does not exist" containerID="f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a" Jul 14 22:20:20.888077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.888018 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a"} err="failed to get container status \"f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a\": rpc error: code = NotFound desc = could not find container \"f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a\": container with ID starting with f5a228c8af9fc92217b713aeee89cdad75eea4c07867d4e0ab1052ec393e128a not found: ID does not exist" Jul 14 22:20:20.899862 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.899837 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r59r8"] Jul 14 22:20:20.902385 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:20.902367 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r59r8"] Jul 14 22:20:22.837042 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:20:22.837010 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d09cf311-d5c1-4a49-9e26-8daed7831644" path="/var/lib/kubelet/pods/d09cf311-d5c1-4a49-9e26-8daed7831644/volumes" Jul 14 22:21:01.256420 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:01.256369 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:21:08.106728 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:08.106696 2578 generic.go:356] "Generic (PLEG): container finished" podID="85c6620a-8dd6-45f2-9862-a34d4e4afe86" containerID="16da43cfb8fce92bf39cca0efc78d2f7136a02363cdd9b0a642342cabc7bb448" exitCode=0 Jul 14 22:21:08.107157 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:08.106765 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" event={"ID":"85c6620a-8dd6-45f2-9862-a34d4e4afe86","Type":"ContainerDied","Data":"16da43cfb8fce92bf39cca0efc78d2f7136a02363cdd9b0a642342cabc7bb448"} Jul 14 22:21:08.107157 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:08.107097 2578 scope.go:117] "RemoveContainer" containerID="16da43cfb8fce92bf39cca0efc78d2f7136a02363cdd9b0a642342cabc7bb448" Jul 14 22:21:08.108100 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:08.108072 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:21:08.108169 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:08.108140 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:21:08.108210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:08.108170 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:21:09.113884 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:09.113848 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" event={"ID":"85c6620a-8dd6-45f2-9862-a34d4e4afe86","Type":"ContainerStarted","Data":"5aa9b8faa7af9d15200ec7b5d2e6b637b57c2bba44a290e97231ccc22434d58c"} Jul 14 22:21:10.085115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:10.085082 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:21:10.086301 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:10.086280 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rjcqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 14 22:21:10.086371 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:10.086327 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" podUID="85c6620a-8dd6-45f2-9862-a34d4e4afe86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 14 22:21:11.085993 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:11.085954 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rjcqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 14 22:21:11.086384 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:11.086040 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" podUID="85c6620a-8dd6-45f2-9862-a34d4e4afe86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 14 22:21:12.085487 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:12.085453 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rjcqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 14 22:21:12.085657 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:12.085512 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" podUID="85c6620a-8dd6-45f2-9862-a34d4e4afe86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 14 22:21:12.685782 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:12.685748 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:21:13.088997 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:13.088971 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:21:13.133090 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:13.133062 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rjcqf" Jul 14 22:21:24.118141 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:24.118101 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:21:38.904590 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:38.904547 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:21:47.051095 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:47.051057 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:21:57.589935 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.589900 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j"] Jul 14 22:21:57.590499 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.590478 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d09cf311-d5c1-4a49-9e26-8daed7831644" containerName="authorino" Jul 14 22:21:57.590593 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.590502 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d09cf311-d5c1-4a49-9e26-8daed7831644" containerName="authorino" Jul 14 22:21:57.590646 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.590611 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d09cf311-d5c1-4a49-9e26-8daed7831644" containerName="authorino" Jul 14 22:21:57.593613 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.593592 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.596839 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.596817 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 22:21:57.598097 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.598079 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 22:21:57.598210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.598082 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 14 22:21:57.598349 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.598317 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lhvtx\"" Jul 14 22:21:57.615131 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.615106 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j"] Jul 14 22:21:57.678718 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.678688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.678850 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.678724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.678850 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.678746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.678850 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.678787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cf5653-385e-48de-9af8-74b4d56f8a0a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.678850 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.678843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2kxb\" (UniqueName: \"kubernetes.io/projected/03cf5653-385e-48de-9af8-74b4d56f8a0a-kube-api-access-b2kxb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.679010 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.678883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.779964 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.779932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.780128 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.779971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.780128 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.779997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.780128 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.780016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cf5653-385e-48de-9af8-74b4d56f8a0a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.780128 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.780053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2kxb\" (UniqueName: \"kubernetes.io/projected/03cf5653-385e-48de-9af8-74b4d56f8a0a-kube-api-access-b2kxb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.780128 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.780072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.780428 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.780403 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.780496 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.780468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.780543 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.780520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.782383 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.782365 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cf5653-385e-48de-9af8-74b4d56f8a0a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.782548 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.782532 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cf5653-385e-48de-9af8-74b4d56f8a0a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.789011 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.788988 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2kxb\" (UniqueName: \"kubernetes.io/projected/03cf5653-385e-48de-9af8-74b4d56f8a0a-kube-api-access-b2kxb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j\" (UID: \"03cf5653-385e-48de-9af8-74b4d56f8a0a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:57.904386 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:57.904309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:21:58.041046 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:58.041016 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j"] Jul 14 22:21:58.041752 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:21:58.041718 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03cf5653_385e_48de_9af8_74b4d56f8a0a.slice/crio-c7a11736e40039366258602f2c45d8feb58ed0412551f47e7ba43a853380789c WatchSource:0}: Error finding container c7a11736e40039366258602f2c45d8feb58ed0412551f47e7ba43a853380789c: Status 404 returned error can't find the container with id c7a11736e40039366258602f2c45d8feb58ed0412551f47e7ba43a853380789c Jul 14 22:21:58.043034 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:58.043016 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:21:58.360298 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:58.360252 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" event={"ID":"03cf5653-385e-48de-9af8-74b4d56f8a0a","Type":"ContainerStarted","Data":"c7a11736e40039366258602f2c45d8feb58ed0412551f47e7ba43a853380789c"} Jul 14 22:21:59.226587 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:59.226552 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:21:59.366213 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:59.366131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" event={"ID":"03cf5653-385e-48de-9af8-74b4d56f8a0a","Type":"ContainerStarted","Data":"db34784e48bda2c8cffa3d2caab68edeb2fadc9841c31f78da0147c5acb2a67e"} Jul 14 22:21:59.401357 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:21:59.401312 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" podStartSLOduration=1.3761355100000001 podStartE2EDuration="2.401296263s" totalImagesPullingTime="1.025160753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:21:59.400848925 +0000 UTC m=+785.213480565" watchObservedRunningTime="2026-07-14 22:21:59.401296263 +0000 UTC m=+785.213927903" Jul 14 22:22:07.905294 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:22:07.905255 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:22:07.905777 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:22:07.905346 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:22:07.919401 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:22:07.919374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:22:08.426071 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:22:08.426043 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j" Jul 14 22:22:10.000328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:22:10.000298 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:22:27.824949 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:22:27.824864 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:23:38.485406 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.485371 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b854c6cf6-xldwt"] Jul 14 22:23:38.488699 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.488682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b854c6cf6-xldwt" Jul 14 22:23:38.499646 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.499622 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b854c6cf6-xldwt"] Jul 14 22:23:38.637205 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.637167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjwf\" (UniqueName: \"kubernetes.io/projected/01d52704-bc50-4e60-acd2-b40efc2b3bd5-kube-api-access-6qjwf\") pod \"authorino-b854c6cf6-xldwt\" (UID: \"01d52704-bc50-4e60-acd2-b40efc2b3bd5\") " pod="kuadrant-system/authorino-b854c6cf6-xldwt" Jul 14 22:23:38.637389 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.637251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/01d52704-bc50-4e60-acd2-b40efc2b3bd5-tls-cert\") pod \"authorino-b854c6cf6-xldwt\" (UID: \"01d52704-bc50-4e60-acd2-b40efc2b3bd5\") " pod="kuadrant-system/authorino-b854c6cf6-xldwt" Jul 14 22:23:38.737970 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.737879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/01d52704-bc50-4e60-acd2-b40efc2b3bd5-tls-cert\") pod \"authorino-b854c6cf6-xldwt\" (UID: \"01d52704-bc50-4e60-acd2-b40efc2b3bd5\") " pod="kuadrant-system/authorino-b854c6cf6-xldwt" Jul 14 22:23:38.737970 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.737948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjwf\" (UniqueName: \"kubernetes.io/projected/01d52704-bc50-4e60-acd2-b40efc2b3bd5-kube-api-access-6qjwf\") pod \"authorino-b854c6cf6-xldwt\" (UID: \"01d52704-bc50-4e60-acd2-b40efc2b3bd5\") " pod="kuadrant-system/authorino-b854c6cf6-xldwt" Jul 14 22:23:38.740292 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.740269 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/01d52704-bc50-4e60-acd2-b40efc2b3bd5-tls-cert\") pod \"authorino-b854c6cf6-xldwt\" (UID: \"01d52704-bc50-4e60-acd2-b40efc2b3bd5\") " pod="kuadrant-system/authorino-b854c6cf6-xldwt" Jul 14 22:23:38.746630 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.746606 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjwf\" (UniqueName: \"kubernetes.io/projected/01d52704-bc50-4e60-acd2-b40efc2b3bd5-kube-api-access-6qjwf\") pod \"authorino-b854c6cf6-xldwt\" (UID: \"01d52704-bc50-4e60-acd2-b40efc2b3bd5\") " pod="kuadrant-system/authorino-b854c6cf6-xldwt" Jul 14 22:23:38.798440 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.798413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b854c6cf6-xldwt" Jul 14 22:23:38.944196 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:38.944170 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b854c6cf6-xldwt"] Jul 14 22:23:38.946523 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:23:38.946491 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01d52704_bc50_4e60_acd2_b40efc2b3bd5.slice/crio-878ea1962ceac0a87fdae7e01d7173173aff768661adffb2f187781938552186 WatchSource:0}: Error finding container 878ea1962ceac0a87fdae7e01d7173173aff768661adffb2f187781938552186: Status 404 returned error can't find the container with id 878ea1962ceac0a87fdae7e01d7173173aff768661adffb2f187781938552186 Jul 14 22:23:39.853682 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:39.853640 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b854c6cf6-xldwt" event={"ID":"01d52704-bc50-4e60-acd2-b40efc2b3bd5","Type":"ContainerStarted","Data":"285e3f7671988f1256bca4045c27e1e4e279a14c1d6bfeaeca6e3400683db42c"} Jul 14 22:23:39.854153 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:39.853693 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b854c6cf6-xldwt" event={"ID":"01d52704-bc50-4e60-acd2-b40efc2b3bd5","Type":"ContainerStarted","Data":"878ea1962ceac0a87fdae7e01d7173173aff768661adffb2f187781938552186"} Jul 14 22:23:39.943894 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:39.943815 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b854c6cf6-xldwt" podStartSLOduration=1.485516173 podStartE2EDuration="1.943774437s" totalImagesPullingTime="458.258264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:23:39.898604093 +0000 UTC m=+885.711235734" watchObservedRunningTime="2026-07-14 22:23:39.943774437 +0000 UTC m=+885.756406078" Jul 14 22:23:39.946448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:39.946420 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85b78c57d9-rq4fm"] Jul 14 22:23:39.946617 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:39.946588 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" podUID="255e7b01-494b-46c0-b862-61369368a576" containerName="authorino" containerID="cri-o://f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582" gracePeriod=30 Jul 14 22:23:40.222517 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.222493 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:23:40.251195 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.251161 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/255e7b01-494b-46c0-b862-61369368a576-tls-cert\") pod \"255e7b01-494b-46c0-b862-61369368a576\" (UID: \"255e7b01-494b-46c0-b862-61369368a576\") " Jul 14 22:23:40.251365 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.251216 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8mds\" (UniqueName: \"kubernetes.io/projected/255e7b01-494b-46c0-b862-61369368a576-kube-api-access-q8mds\") pod \"255e7b01-494b-46c0-b862-61369368a576\" (UID: \"255e7b01-494b-46c0-b862-61369368a576\") " Jul 14 22:23:40.253388 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.253355 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255e7b01-494b-46c0-b862-61369368a576-kube-api-access-q8mds" (OuterVolumeSpecName: "kube-api-access-q8mds") pod "255e7b01-494b-46c0-b862-61369368a576" (UID: "255e7b01-494b-46c0-b862-61369368a576"). InnerVolumeSpecName "kube-api-access-q8mds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:23:40.264075 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.264053 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/255e7b01-494b-46c0-b862-61369368a576-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "255e7b01-494b-46c0-b862-61369368a576" (UID: "255e7b01-494b-46c0-b862-61369368a576"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:23:40.352069 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.352027 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/255e7b01-494b-46c0-b862-61369368a576-tls-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:23:40.352069 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.352070 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8mds\" (UniqueName: \"kubernetes.io/projected/255e7b01-494b-46c0-b862-61369368a576-kube-api-access-q8mds\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:23:40.859584 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.859546 2578 generic.go:356] "Generic (PLEG): container finished" podID="255e7b01-494b-46c0-b862-61369368a576" containerID="f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582" exitCode=0 Jul 14 22:23:40.860006 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.859590 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" Jul 14 22:23:40.860006 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.859627 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" event={"ID":"255e7b01-494b-46c0-b862-61369368a576","Type":"ContainerDied","Data":"f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582"} Jul 14 22:23:40.860006 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.859668 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85b78c57d9-rq4fm" event={"ID":"255e7b01-494b-46c0-b862-61369368a576","Type":"ContainerDied","Data":"3a61445a583f214656576b56cfc078dd62edc9fc0ee7691fcd79443141edb34f"} Jul 14 22:23:40.860006 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.859690 2578 scope.go:117] "RemoveContainer" containerID="f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582" Jul 14 22:23:40.869431 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.869413 2578 scope.go:117] "RemoveContainer" containerID="f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582" Jul 14 22:23:40.869730 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:23:40.869712 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582\": container with ID starting with f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582 not found: ID does not exist" containerID="f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582" Jul 14 22:23:40.869811 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.869742 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582"} err="failed to get container status \"f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582\": rpc error: code = NotFound desc = could not find container \"f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582\": container with ID starting with f4275433da534fd61a0ad8daf13589dd20de89a89346e1360a6e5d8e95bf0582 not found: ID does not exist" Jul 14 22:23:40.892347 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.892319 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85b78c57d9-rq4fm"] Jul 14 22:23:40.904927 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:40.904895 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85b78c57d9-rq4fm"] Jul 14 22:23:42.838200 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:42.838170 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="255e7b01-494b-46c0-b862-61369368a576" path="/var/lib/kubelet/pods/255e7b01-494b-46c0-b862-61369368a576/volumes" Jul 14 22:23:54.682601 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:54.682528 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:23:54.683029 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:23:54.682727 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:24:13.742931 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:24:13.742894 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:24:23.043796 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:24:23.043754 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:24:33.994520 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:24:33.994483 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:24:43.205977 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:24:43.205943 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:24:54.201079 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:24:54.201041 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:25:21.120303 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:25:21.120211 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:25:34.100911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:25:34.100877 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:26:10.211565 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:26:10.211525 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:26:20.010853 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:26:20.010816 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:26:29.999873 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:26:29.999837 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:26:37.498552 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:26:37.498520 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:26:47.596576 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:26:47.596386 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:27:02.524568 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:27:02.524536 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:27:09.804832 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:27:09.804791 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:27:17.516264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:27:17.516207 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:27:42.634585 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:27:42.634553 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:27:52.694049 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:27:52.694012 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:28:22.405099 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:28:22.404998 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:28:33.308704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:28:33.308671 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:28:48.793757 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:28:48.793719 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:28:54.714966 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:28:54.714936 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:28:54.715538 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:28:54.715178 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:28:58.203326 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:28:58.203292 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:29:08.126666 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:08.126633 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:29:17.507722 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:17.507687 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:29:26.825339 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:26.825301 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:29:53.523793 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:53.523704 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6"] Jul 14 22:29:53.524323 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:53.523890 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" podUID="6ea6547b-2bf0-49c1-9664-ac4d8cc525aa" containerName="manager" containerID="cri-o://3eafbcd5139f715705acb4f3915fadc0010b0257a5a4a48d908cb7363b6bc6ac" gracePeriod=10 Jul 14 22:29:56.604526 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:56.604486 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" podUID="6ea6547b-2bf0-49c1-9664-ac4d8cc525aa" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Jul 14 22:29:56.693892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:56.693803 2578 generic.go:356] "Generic (PLEG): container finished" podID="6ea6547b-2bf0-49c1-9664-ac4d8cc525aa" containerID="3eafbcd5139f715705acb4f3915fadc0010b0257a5a4a48d908cb7363b6bc6ac" exitCode=0 Jul 14 22:29:56.693892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:56.693865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" event={"ID":"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa","Type":"ContainerDied","Data":"3eafbcd5139f715705acb4f3915fadc0010b0257a5a4a48d908cb7363b6bc6ac"} Jul 14 22:29:57.075559 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.075533 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:29:57.247067 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.246972 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69988\" (UniqueName: \"kubernetes.io/projected/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-kube-api-access-69988\") pod \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\" (UID: \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\") " Jul 14 22:29:57.247067 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.247029 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-extensions-socket-volume\") pod \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\" (UID: \"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa\") " Jul 14 22:29:57.247437 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.247415 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6ea6547b-2bf0-49c1-9664-ac4d8cc525aa" (UID: "6ea6547b-2bf0-49c1-9664-ac4d8cc525aa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:29:57.249098 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.249073 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-kube-api-access-69988" (OuterVolumeSpecName: "kube-api-access-69988") pod "6ea6547b-2bf0-49c1-9664-ac4d8cc525aa" (UID: "6ea6547b-2bf0-49c1-9664-ac4d8cc525aa"). InnerVolumeSpecName "kube-api-access-69988". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:29:57.347706 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.347681 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69988\" (UniqueName: \"kubernetes.io/projected/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-kube-api-access-69988\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:29:57.347825 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.347724 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa-extensions-socket-volume\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:29:57.700351 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.700323 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" Jul 14 22:29:57.700759 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.700324 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6" event={"ID":"6ea6547b-2bf0-49c1-9664-ac4d8cc525aa","Type":"ContainerDied","Data":"9da9c9e6579817bec3c286b23afb7222df34a37e32c31ba117a51d7ec7e057b0"} Jul 14 22:29:57.700759 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.700460 2578 scope.go:117] "RemoveContainer" containerID="3eafbcd5139f715705acb4f3915fadc0010b0257a5a4a48d908cb7363b6bc6ac" Jul 14 22:29:57.722803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.722777 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6"] Jul 14 22:29:57.726617 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:57.726597 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8rtl6"] Jul 14 22:29:58.837353 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:29:58.837317 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea6547b-2bf0-49c1-9664-ac4d8cc525aa" path="/var/lib/kubelet/pods/6ea6547b-2bf0-49c1-9664-ac4d8cc525aa/volumes" Jul 14 22:30:00.182732 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.182698 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps"] Jul 14 22:30:00.183102 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.183045 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea6547b-2bf0-49c1-9664-ac4d8cc525aa" containerName="manager" Jul 14 22:30:00.183102 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.183055 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea6547b-2bf0-49c1-9664-ac4d8cc525aa" containerName="manager" Jul 14 22:30:00.183102 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.183069 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="255e7b01-494b-46c0-b862-61369368a576" containerName="authorino" Jul 14 22:30:00.183102 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.183074 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="255e7b01-494b-46c0-b862-61369368a576" containerName="authorino" Jul 14 22:30:00.183270 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.183143 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="255e7b01-494b-46c0-b862-61369368a576" containerName="authorino" Jul 14 22:30:00.183270 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.183151 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea6547b-2bf0-49c1-9664-ac4d8cc525aa" containerName="manager" Jul 14 22:30:00.187575 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.187558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps" Jul 14 22:30:00.190766 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.190744 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:30:00.197945 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.197919 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps"] Jul 14 22:30:00.198465 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.198447 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps" Jul 14 22:30:00.329771 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.329746 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps"] Jul 14 22:30:00.332406 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:30:00.332378 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675a968d_e550_46c8_aa15_f5a4b3d9eb8e.slice/crio-a9eb86d65ad4a3dfc3724c1dec60dc804fc4d6c03dafc6499f8c3bd776e19413 WatchSource:0}: Error finding container a9eb86d65ad4a3dfc3724c1dec60dc804fc4d6c03dafc6499f8c3bd776e19413: Status 404 returned error can't find the container with id a9eb86d65ad4a3dfc3724c1dec60dc804fc4d6c03dafc6499f8c3bd776e19413 Jul 14 22:30:00.334313 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.334295 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:30:00.719078 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:00.719044 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps" event={"ID":"675a968d-e550-46c8-aa15-f5a4b3d9eb8e","Type":"ContainerStarted","Data":"a9eb86d65ad4a3dfc3724c1dec60dc804fc4d6c03dafc6499f8c3bd776e19413"} Jul 14 22:30:02.730103 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:02.730064 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps" event={"ID":"675a968d-e550-46c8-aa15-f5a4b3d9eb8e","Type":"ContainerStarted","Data":"c11ba52f4ef2a7fc01f6165f4e0e0a6d0b35ec49013f741d7b6b75791a177105"} Jul 14 22:30:02.752042 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:02.751995 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps" podStartSLOduration=0.781230642 podStartE2EDuration="2.751979567s" totalImagesPullingTime="1.970748925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:30:02.749196031 +0000 UTC m=+1268.561827687" watchObservedRunningTime="2026-07-14 22:30:02.751979567 +0000 UTC m=+1268.564611206" Jul 14 22:30:03.736855 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:03.736815 2578 generic.go:356] "Generic (PLEG): container finished" podID="675a968d-e550-46c8-aa15-f5a4b3d9eb8e" containerID="c11ba52f4ef2a7fc01f6165f4e0e0a6d0b35ec49013f741d7b6b75791a177105" exitCode=0 Jul 14 22:30:03.737235 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:03.736907 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps" event={"ID":"675a968d-e550-46c8-aa15-f5a4b3d9eb8e","Type":"ContainerDied","Data":"c11ba52f4ef2a7fc01f6165f4e0e0a6d0b35ec49013f741d7b6b75791a177105"} Jul 14 22:30:04.872972 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:04.872948 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps" Jul 14 22:30:05.748352 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:05.748324 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps" Jul 14 22:30:05.748529 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:05.748360 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734470-xmfps" event={"ID":"675a968d-e550-46c8-aa15-f5a4b3d9eb8e","Type":"ContainerDied","Data":"a9eb86d65ad4a3dfc3724c1dec60dc804fc4d6c03dafc6499f8c3bd776e19413"} Jul 14 22:30:05.748529 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:05.748405 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9eb86d65ad4a3dfc3724c1dec60dc804fc4d6c03dafc6499f8c3bd776e19413" Jul 14 22:30:54.766034 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.766002 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml"] Jul 14 22:30:54.766528 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.766401 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675a968d-e550-46c8-aa15-f5a4b3d9eb8e" containerName="cleanup" Jul 14 22:30:54.766528 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.766413 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="675a968d-e550-46c8-aa15-f5a4b3d9eb8e" containerName="cleanup" Jul 14 22:30:54.766528 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.766492 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="675a968d-e550-46c8-aa15-f5a4b3d9eb8e" containerName="cleanup" Jul 14 22:30:54.769660 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.769640 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:30:54.774631 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.774607 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4n2r\"" Jul 14 22:30:54.784430 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.784410 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml"] Jul 14 22:30:54.843737 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.843708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f4e2a3d-e431-4da3-b171-e798a6afa8e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-89dml\" (UID: \"7f4e2a3d-e431-4da3-b171-e798a6afa8e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:30:54.843893 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.843748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv5zt\" (UniqueName: \"kubernetes.io/projected/7f4e2a3d-e431-4da3-b171-e798a6afa8e9-kube-api-access-gv5zt\") pod \"kuadrant-operator-controller-manager-86878c769f-89dml\" (UID: \"7f4e2a3d-e431-4da3-b171-e798a6afa8e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:30:54.944232 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.944188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f4e2a3d-e431-4da3-b171-e798a6afa8e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-89dml\" (UID: \"7f4e2a3d-e431-4da3-b171-e798a6afa8e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:30:54.944440 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.944272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv5zt\" (UniqueName: \"kubernetes.io/projected/7f4e2a3d-e431-4da3-b171-e798a6afa8e9-kube-api-access-gv5zt\") pod \"kuadrant-operator-controller-manager-86878c769f-89dml\" (UID: \"7f4e2a3d-e431-4da3-b171-e798a6afa8e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:30:54.944730 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.944709 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f4e2a3d-e431-4da3-b171-e798a6afa8e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-89dml\" (UID: \"7f4e2a3d-e431-4da3-b171-e798a6afa8e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:30:54.957881 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:54.957846 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv5zt\" (UniqueName: \"kubernetes.io/projected/7f4e2a3d-e431-4da3-b171-e798a6afa8e9-kube-api-access-gv5zt\") pod \"kuadrant-operator-controller-manager-86878c769f-89dml\" (UID: \"7f4e2a3d-e431-4da3-b171-e798a6afa8e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:30:55.079472 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:55.079437 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:30:55.225976 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:55.225873 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml"] Jul 14 22:30:55.230164 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:30:55.230137 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4e2a3d_e431_4da3_b171_e798a6afa8e9.slice/crio-8867c6b1bcef9e53e9e3eaaf7c77154f704a7f55cd880bf95650b96bc18d5cd9 WatchSource:0}: Error finding container 8867c6b1bcef9e53e9e3eaaf7c77154f704a7f55cd880bf95650b96bc18d5cd9: Status 404 returned error can't find the container with id 8867c6b1bcef9e53e9e3eaaf7c77154f704a7f55cd880bf95650b96bc18d5cd9 Jul 14 22:30:55.986712 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:55.986679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" event={"ID":"7f4e2a3d-e431-4da3-b171-e798a6afa8e9","Type":"ContainerStarted","Data":"04578cd7ba05c0d0449d7ef87576d15ec3af07f63a8ee7bb7ad68261d5cdefda"} Jul 14 22:30:55.987118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:55.986734 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" event={"ID":"7f4e2a3d-e431-4da3-b171-e798a6afa8e9","Type":"ContainerStarted","Data":"8867c6b1bcef9e53e9e3eaaf7c77154f704a7f55cd880bf95650b96bc18d5cd9"} Jul 14 22:30:55.987118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:55.986766 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:30:56.008930 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:30:56.008875 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" podStartSLOduration=2.008854091 podStartE2EDuration="2.008854091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:30:56.006868017 +0000 UTC m=+1321.819499656" watchObservedRunningTime="2026-07-14 22:30:56.008854091 +0000 UTC m=+1321.821485732" Jul 14 22:31:06.993120 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:31:06.993088 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-89dml" Jul 14 22:31:16.812455 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:31:16.812413 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:31:29.687789 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:31:29.687750 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:31:38.892786 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:31:38.892748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:31:48.589168 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:31:48.589133 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:31:59.490686 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:31:59.490645 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:32:07.526518 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:32:07.526472 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:32:18.791638 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:32:18.791604 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:32:29.788034 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:32:29.787989 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:32:37.683299 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:32:37.683267 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:32:48.696262 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:32:48.696143 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:33:01.591933 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:33:01.591894 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:33:10.498159 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:33:10.498126 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:33:31.686545 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:33:31.686510 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:33:39.098159 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:33:39.098122 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:33:48.491840 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:33:48.491797 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:33:54.745378 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:33:54.745345 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:33:54.746291 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:33:54.746267 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:33:58.199434 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:33:58.199407 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:34:19.292047 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:34:19.291950 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:34:28.888542 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:34:28.888504 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:34:38.203336 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:34:38.203226 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:34:48.894563 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:34:48.894527 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:34:58.710525 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:34:58.710490 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:35:08.690641 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:35:08.690604 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:35:19.195281 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:35:19.195232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:35:31.394219 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:35:31.394183 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:35:41.010511 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:35:41.010475 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:35:53.425673 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:35:53.425593 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:36:05.809305 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:36:05.809273 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:36:13.699665 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:36:13.699633 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:36:25.701464 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:36:25.701423 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:36:37.603425 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:36:37.603389 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:36:51.608376 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:36:51.608341 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:37:04.001250 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:04.001190 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:37:13.395773 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:13.395735 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:37:24.050922 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.050847 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw"] Jul 14 22:37:24.054997 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.054977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.059804 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.059781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2b6fcd6d-openshift-default-dockercfg-d227w\"" Jul 14 22:37:24.081479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.081458 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw"] Jul 14 22:37:24.156696 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.156659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-podinfo\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.156696 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.156695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istiod-ca-cert\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.156892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.156730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-socket\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.156892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.156753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-credential-socket\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.156892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.156794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-certs\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.156892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.156865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-data\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.157018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.156912 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-envoy\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.157018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.156935 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-token\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.157018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.156967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzg7n\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-kube-api-access-dzg7n\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.257931 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.257891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-token\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258119 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.257953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzg7n\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-kube-api-access-dzg7n\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258119 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-podinfo\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258119 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istiod-ca-cert\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258119 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-socket\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258119 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-credential-socket\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258422 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-certs\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258422 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-data\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258422 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-envoy\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258605 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258555 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-socket\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258738 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-credential-socket\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258738 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258672 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-certs\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258738 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258732 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-data\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.258947 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.258925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istiod-ca-cert\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.260459 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.260440 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-envoy\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.260560 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.260473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-podinfo\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.267265 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.267222 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-token\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.267413 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.267396 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzg7n\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-kube-api-access-dzg7n\") pod \"e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.367966 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.367939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:24.531486 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.531449 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw"] Jul 14 22:37:24.535427 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:37:24.535396 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816017c9_ff3a_4360_bda6_fe71bbe8a2ef.slice/crio-0b39182e8a9549eb8fedf27df210b232293716030dc891d2d9398cc2206ae147 WatchSource:0}: Error finding container 0b39182e8a9549eb8fedf27df210b232293716030dc891d2d9398cc2206ae147: Status 404 returned error can't find the container with id 0b39182e8a9549eb8fedf27df210b232293716030dc891d2d9398cc2206ae147 Jul 14 22:37:24.537843 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.537801 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:37:24.537943 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.537892 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:37:24.537984 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.537951 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:37:24.628089 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.628019 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:37:24.936975 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.936871 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" event={"ID":"816017c9-ff3a-4360-bda6-fe71bbe8a2ef","Type":"ContainerStarted","Data":"25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0"} Jul 14 22:37:24.936975 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.936934 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" event={"ID":"816017c9-ff3a-4360-bda6-fe71bbe8a2ef","Type":"ContainerStarted","Data":"0b39182e8a9549eb8fedf27df210b232293716030dc891d2d9398cc2206ae147"} Jul 14 22:37:24.962776 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:24.962732 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" podStartSLOduration=0.962710571 podStartE2EDuration="962.710571ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:37:24.962002398 +0000 UTC m=+1710.774634038" watchObservedRunningTime="2026-07-14 22:37:24.962710571 +0000 UTC m=+1710.775342211" Jul 14 22:37:25.368370 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:25.368337 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:25.373403 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:25.373379 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:25.942165 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:25.942127 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:25.942966 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:25.942946 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:30.308844 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:30.308794 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh"] Jul 14 22:37:30.312789 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:30.312765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" Jul 14 22:37:30.315594 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:30.315571 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:37:30.315715 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:30.315697 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:37:30.320161 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:30.320137 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh"] Jul 14 22:37:30.413868 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:30.413831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97429e91-3cf1-46db-9a32-4c7eff01843a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh\" (UID: \"97429e91-3cf1-46db-9a32-4c7eff01843a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" Jul 14 22:37:30.514418 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:30.514379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97429e91-3cf1-46db-9a32-4c7eff01843a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh\" (UID: \"97429e91-3cf1-46db-9a32-4c7eff01843a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" Jul 14 22:37:30.515020 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:30.515002 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97429e91-3cf1-46db-9a32-4c7eff01843a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh\" (UID: \"97429e91-3cf1-46db-9a32-4c7eff01843a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" Jul 14 22:37:30.626595 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:30.626500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" Jul 14 22:37:31.024499 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:31.018724 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh"] Jul 14 22:37:31.026363 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:37:31.026338 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97429e91_3cf1_46db_9a32_4c7eff01843a.slice/crio-42ec01e1f1769569cbf9b079dad3616edbb432557db9866b59b1a21e3fe5846a WatchSource:0}: Error finding container 42ec01e1f1769569cbf9b079dad3616edbb432557db9866b59b1a21e3fe5846a: Status 404 returned error can't find the container with id 42ec01e1f1769569cbf9b079dad3616edbb432557db9866b59b1a21e3fe5846a Jul 14 22:37:31.973938 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:31.973894 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" event={"ID":"97429e91-3cf1-46db-9a32-4c7eff01843a","Type":"ContainerStarted","Data":"b41c8d176684621dd5893127f72154855354267fc6613b5ed4d015bbfdff289c"} Jul 14 22:37:31.974367 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:31.973961 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" event={"ID":"97429e91-3cf1-46db-9a32-4c7eff01843a","Type":"ContainerStarted","Data":"42ec01e1f1769569cbf9b079dad3616edbb432557db9866b59b1a21e3fe5846a"} Jul 14 22:37:31.992920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:31.992871 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" podStartSLOduration=1.9928556899999998 podStartE2EDuration="1.99285569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:37:31.991718681 +0000 UTC m=+1717.804350320" watchObservedRunningTime="2026-07-14 22:37:31.99285569 +0000 UTC m=+1717.805487330" Jul 14 22:37:32.979690 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:32.979651 2578 generic.go:356] "Generic (PLEG): container finished" podID="97429e91-3cf1-46db-9a32-4c7eff01843a" containerID="b41c8d176684621dd5893127f72154855354267fc6613b5ed4d015bbfdff289c" exitCode=7 Jul 14 22:37:32.980141 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:32.979738 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" event={"ID":"97429e91-3cf1-46db-9a32-4c7eff01843a","Type":"ContainerDied","Data":"b41c8d176684621dd5893127f72154855354267fc6613b5ed4d015bbfdff289c"} Jul 14 22:37:32.980141 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:32.979888 2578 scope.go:117] "RemoveContainer" containerID="b41c8d176684621dd5893127f72154855354267fc6613b5ed4d015bbfdff289c" Jul 14 22:37:33.986154 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:33.986123 2578 generic.go:356] "Generic (PLEG): container finished" podID="97429e91-3cf1-46db-9a32-4c7eff01843a" containerID="05b732fa826c85ce881d4ad313aa5f526156a89e6977c2505855d4ff30aa7057" exitCode=0 Jul 14 22:37:33.986524 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:33.986210 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" event={"ID":"97429e91-3cf1-46db-9a32-4c7eff01843a","Type":"ContainerDied","Data":"05b732fa826c85ce881d4ad313aa5f526156a89e6977c2505855d4ff30aa7057"} Jul 14 22:37:33.986524 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:33.986309 2578 scope.go:117] "RemoveContainer" containerID="b41c8d176684621dd5893127f72154855354267fc6613b5ed4d015bbfdff289c" Jul 14 22:37:35.138099 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:35.138071 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" Jul 14 22:37:35.261679 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:35.261589 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97429e91-3cf1-46db-9a32-4c7eff01843a-maas-api-service-ca\") pod \"97429e91-3cf1-46db-9a32-4c7eff01843a\" (UID: \"97429e91-3cf1-46db-9a32-4c7eff01843a\") " Jul 14 22:37:35.262035 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:35.262010 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97429e91-3cf1-46db-9a32-4c7eff01843a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97429e91-3cf1-46db-9a32-4c7eff01843a" (UID: "97429e91-3cf1-46db-9a32-4c7eff01843a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:37:35.363154 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:35.363109 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97429e91-3cf1-46db-9a32-4c7eff01843a-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:37:35.999575 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:35.999535 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" event={"ID":"97429e91-3cf1-46db-9a32-4c7eff01843a","Type":"ContainerDied","Data":"42ec01e1f1769569cbf9b079dad3616edbb432557db9866b59b1a21e3fe5846a"} Jul 14 22:37:35.999767 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:35.999589 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh" Jul 14 22:37:35.999767 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:35.999598 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ec01e1f1769569cbf9b079dad3616edbb432557db9866b59b1a21e3fe5846a" Jul 14 22:37:36.999205 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:36.999169 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:37:38.010723 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:38.010689 2578 generic.go:356] "Generic (PLEG): container finished" podID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerID="25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0" exitCode=0 Jul 14 22:37:38.011235 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:38.010764 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" event={"ID":"816017c9-ff3a-4360-bda6-fe71bbe8a2ef","Type":"ContainerDied","Data":"25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0"} Jul 14 22:37:38.011333 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:38.011256 2578 scope.go:117] "RemoveContainer" containerID="25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0" Jul 14 22:37:38.012225 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:38.012198 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:37:38.012302 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:38.012289 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:37:38.012348 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:38.012321 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:37:39.018171 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:39.018135 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" event={"ID":"816017c9-ff3a-4360-bda6-fe71bbe8a2ef","Type":"ContainerStarted","Data":"5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb"} Jul 14 22:37:39.368436 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:39.368392 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:39.369697 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:39.369663 2578 patch_prober.go:28] interesting pod/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 22:37:39.369851 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:39.369719 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 22:37:40.368872 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:40.368837 2578 patch_prober.go:28] interesting pod/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 22:37:40.369324 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:40.368901 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 22:37:40.942720 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:40.942682 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:41.368328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:41.368290 2578 patch_prober.go:28] interesting pod/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 22:37:41.368533 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:41.368343 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 22:37:42.369181 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:42.369143 2578 patch_prober.go:28] interesting pod/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 22:37:42.369587 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:42.369228 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 22:37:43.372394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:43.372363 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:43.373325 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:43.373307 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:37:49.092901 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:49.092862 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:37:55.580887 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:55.580854 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw"] Jul 14 22:37:55.581343 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:55.581057 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" containerID="cri-o://5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb" gracePeriod=30 Jul 14 22:37:56.154562 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.154534 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p"] Jul 14 22:37:56.154906 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.154895 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97429e91-3cf1-46db-9a32-4c7eff01843a" containerName="revoke-keys" Jul 14 22:37:56.154957 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.154908 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97429e91-3cf1-46db-9a32-4c7eff01843a" containerName="revoke-keys" Jul 14 22:37:56.154957 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.154922 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97429e91-3cf1-46db-9a32-4c7eff01843a" containerName="revoke-keys" Jul 14 22:37:56.154957 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.154927 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97429e91-3cf1-46db-9a32-4c7eff01843a" containerName="revoke-keys" Jul 14 22:37:56.155096 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.155005 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97429e91-3cf1-46db-9a32-4c7eff01843a" containerName="revoke-keys" Jul 14 22:37:56.155096 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.155013 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97429e91-3cf1-46db-9a32-4c7eff01843a" containerName="revoke-keys" Jul 14 22:37:56.158438 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.158420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.162141 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.162122 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-91e25165-gw-openshift-default-dockercfg-w59jz\"" Jul 14 22:37:56.170889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.170864 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p"] Jul 14 22:37:56.243023 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.242998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-credential-socket\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.243144 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.243049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51e3584-46a5-4665-b745-b07b985852b3-istiod-ca-cert\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.243144 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.243066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-certs\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.243144 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.243120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-data\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.243314 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.243175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-envoy\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.243314 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.243287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvlz4\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-kube-api-access-cvlz4\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.243395 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.243317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-socket\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.243395 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.243349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51e3584-46a5-4665-b745-b07b985852b3-istio-podinfo\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.243395 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.243373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-istio-token\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.344666 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.344624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvlz4\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-kube-api-access-cvlz4\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.344666 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.344670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-socket\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.344916 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.344731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51e3584-46a5-4665-b745-b07b985852b3-istio-podinfo\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.344916 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.344766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-istio-token\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.344916 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.344897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-credential-socket\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.345086 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.344983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51e3584-46a5-4665-b745-b07b985852b3-istiod-ca-cert\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.345086 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.345011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-certs\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.345086 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.345030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-data\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.345086 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.345072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-envoy\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.345315 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.345111 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-socket\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.345315 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.345305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-credential-socket\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.345452 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.345428 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-certs\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.345563 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.345484 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-data\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.345764 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.345742 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51e3584-46a5-4665-b745-b07b985852b3-istiod-ca-cert\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.347264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.347231 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-envoy\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.347681 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.347664 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51e3584-46a5-4665-b745-b07b985852b3-istio-podinfo\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.353629 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.353609 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvlz4\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-kube-api-access-cvlz4\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.354231 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.354210 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-istio-token\") pod \"e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.468849 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.468766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:56.613370 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.613347 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p"] Jul 14 22:37:56.615100 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:37:56.615071 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51e3584_46a5_4665_b745_b07b985852b3.slice/crio-430b99e08bd9cea8363fe95c1d391955aabc5349444fccafd8bb127f8aa7d949 WatchSource:0}: Error finding container 430b99e08bd9cea8363fe95c1d391955aabc5349444fccafd8bb127f8aa7d949: Status 404 returned error can't find the container with id 430b99e08bd9cea8363fe95c1d391955aabc5349444fccafd8bb127f8aa7d949 Jul 14 22:37:56.617443 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.617406 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:37:56.617538 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.617471 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:37:56.617538 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:56.617496 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:37:57.112280 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:57.112236 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" event={"ID":"a51e3584-46a5-4665-b745-b07b985852b3","Type":"ContainerStarted","Data":"6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35"} Jul 14 22:37:57.112444 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:57.112295 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" event={"ID":"a51e3584-46a5-4665-b745-b07b985852b3","Type":"ContainerStarted","Data":"430b99e08bd9cea8363fe95c1d391955aabc5349444fccafd8bb127f8aa7d949"} Jul 14 22:37:57.147127 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:57.147076 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" podStartSLOduration=1.147061827 podStartE2EDuration="1.147061827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:37:57.146844804 +0000 UTC m=+1742.959476444" watchObservedRunningTime="2026-07-14 22:37:57.147061827 +0000 UTC m=+1742.959693466" Jul 14 22:37:57.469313 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:57.469217 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:57.474425 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:57.474400 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:58.117744 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:58.117712 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:58.118757 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:58.118740 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:37:58.375171 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:58.375091 2578 patch_prober.go:28] interesting pod/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:37:58.375171 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:37:58.375140 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:38:00.831817 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.831781 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:38:00.991859 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.991827 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-certs\") pod \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " Jul 14 22:38:00.992013 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.991876 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzg7n\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-kube-api-access-dzg7n\") pod \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " Jul 14 22:38:00.992013 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.991946 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-socket\") pod \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " Jul 14 22:38:00.992013 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.991981 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-data\") pod \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " Jul 14 22:38:00.992013 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992011 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-token\") pod \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " Jul 14 22:38:00.992174 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992076 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-envoy\") pod \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " Jul 14 22:38:00.992174 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992101 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-podinfo\") pod \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " Jul 14 22:38:00.992174 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992129 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istiod-ca-cert\") pod \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " Jul 14 22:38:00.992174 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992148 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "816017c9-ff3a-4360-bda6-fe71bbe8a2ef" (UID: "816017c9-ff3a-4360-bda6-fe71bbe8a2ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:00.992423 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992205 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-credential-socket\") pod \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\" (UID: \"816017c9-ff3a-4360-bda6-fe71bbe8a2ef\") " Jul 14 22:38:00.992579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992547 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:00.992579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992555 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "816017c9-ff3a-4360-bda6-fe71bbe8a2ef" (UID: "816017c9-ff3a-4360-bda6-fe71bbe8a2ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:38:00.992723 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992570 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "816017c9-ff3a-4360-bda6-fe71bbe8a2ef" (UID: "816017c9-ff3a-4360-bda6-fe71bbe8a2ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:00.992723 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.992678 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "816017c9-ff3a-4360-bda6-fe71bbe8a2ef" (UID: "816017c9-ff3a-4360-bda6-fe71bbe8a2ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:00.993202 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.993182 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "816017c9-ff3a-4360-bda6-fe71bbe8a2ef" (UID: "816017c9-ff3a-4360-bda6-fe71bbe8a2ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:00.994288 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.994230 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "816017c9-ff3a-4360-bda6-fe71bbe8a2ef" (UID: "816017c9-ff3a-4360-bda6-fe71bbe8a2ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:38:00.994399 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.994312 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "816017c9-ff3a-4360-bda6-fe71bbe8a2ef" (UID: "816017c9-ff3a-4360-bda6-fe71bbe8a2ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:38:00.994515 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.994494 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-kube-api-access-dzg7n" (OuterVolumeSpecName: "kube-api-access-dzg7n") pod "816017c9-ff3a-4360-bda6-fe71bbe8a2ef" (UID: "816017c9-ff3a-4360-bda6-fe71bbe8a2ef"). InnerVolumeSpecName "kube-api-access-dzg7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:38:00.994613 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:00.994554 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "816017c9-ff3a-4360-bda6-fe71bbe8a2ef" (UID: "816017c9-ff3a-4360-bda6-fe71bbe8a2ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:38:01.093070 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.093039 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:01.093070 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.093064 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzg7n\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-kube-api-access-dzg7n\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:01.093070 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.093075 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:01.093317 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.093084 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:01.093317 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.093093 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:01.093317 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.093100 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:01.093317 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.093108 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:01.093317 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.093116 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816017c9-ff3a-4360-bda6-fe71bbe8a2ef-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:01.140403 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.140364 2578 generic.go:356] "Generic (PLEG): container finished" podID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerID="5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb" exitCode=0 Jul 14 22:38:01.140562 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.140411 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" Jul 14 22:38:01.140562 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.140442 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" event={"ID":"816017c9-ff3a-4360-bda6-fe71bbe8a2ef","Type":"ContainerDied","Data":"5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb"} Jul 14 22:38:01.140562 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.140493 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw" event={"ID":"816017c9-ff3a-4360-bda6-fe71bbe8a2ef","Type":"ContainerDied","Data":"0b39182e8a9549eb8fedf27df210b232293716030dc891d2d9398cc2206ae147"} Jul 14 22:38:01.140562 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.140514 2578 scope.go:117] "RemoveContainer" containerID="5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb" Jul 14 22:38:01.150400 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.150382 2578 scope.go:117] "RemoveContainer" containerID="25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0" Jul 14 22:38:01.158617 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.158598 2578 scope.go:117] "RemoveContainer" containerID="5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb" Jul 14 22:38:01.158866 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:38:01.158850 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb\": container with ID starting with 5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb not found: ID does not exist" containerID="5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb" Jul 14 22:38:01.158918 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.158872 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb"} err="failed to get container status \"5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb\": rpc error: code = NotFound desc = could not find container \"5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb\": container with ID starting with 5bc2676ad08309faf48e7e0a68f90483d72ac07dc68e429065c2c2ae6da201eb not found: ID does not exist" Jul 14 22:38:01.158918 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.158887 2578 scope.go:117] "RemoveContainer" containerID="25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0" Jul 14 22:38:01.159106 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:38:01.159088 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0\": container with ID starting with 25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0 not found: ID does not exist" containerID="25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0" Jul 14 22:38:01.159165 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.159116 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0"} err="failed to get container status \"25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0\": rpc error: code = NotFound desc = could not find container \"25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0\": container with ID starting with 25b7926608400e84e3ac669484734560caaf94639a62824f28e0c10fd540d6c0 not found: ID does not exist" Jul 14 22:38:01.163745 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.163722 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw"] Jul 14 22:38:01.167966 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.167947 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2b6fcd6d-openshift-default-66566b68c7-62psw"] Jul 14 22:38:01.963907 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.963875 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr"] Jul 14 22:38:01.964701 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.964676 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" Jul 14 22:38:01.964701 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.964701 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" Jul 14 22:38:01.964882 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.964743 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" Jul 14 22:38:01.964882 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.964753 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" Jul 14 22:38:01.964882 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.964860 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" Jul 14 22:38:01.964882 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.964871 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" containerName="istio-proxy" Jul 14 22:38:01.969778 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.969755 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" Jul 14 22:38:01.973255 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.972943 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:38:01.973387 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.973350 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:38:01.975563 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:01.975476 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr"] Jul 14 22:38:02.105217 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:02.105184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdb621-4037-4a14-bb35-fb86912e1f05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr\" (UID: \"c3bdb621-4037-4a14-bb35-fb86912e1f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" Jul 14 22:38:02.206711 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:02.206675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdb621-4037-4a14-bb35-fb86912e1f05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr\" (UID: \"c3bdb621-4037-4a14-bb35-fb86912e1f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" Jul 14 22:38:02.207307 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:02.207290 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdb621-4037-4a14-bb35-fb86912e1f05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr\" (UID: \"c3bdb621-4037-4a14-bb35-fb86912e1f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" Jul 14 22:38:02.285265 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:02.285130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" Jul 14 22:38:02.627103 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:02.627079 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr"] Jul 14 22:38:02.629376 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:38:02.629348 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3bdb621_4037_4a14_bb35_fb86912e1f05.slice/crio-3e26890146856ecb07896905df0bd2bf2a46b4346d68afeaa3b29deaf68e7f64 WatchSource:0}: Error finding container 3e26890146856ecb07896905df0bd2bf2a46b4346d68afeaa3b29deaf68e7f64: Status 404 returned error can't find the container with id 3e26890146856ecb07896905df0bd2bf2a46b4346d68afeaa3b29deaf68e7f64 Jul 14 22:38:02.839279 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:02.839231 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="816017c9-ff3a-4360-bda6-fe71bbe8a2ef" path="/var/lib/kubelet/pods/816017c9-ff3a-4360-bda6-fe71bbe8a2ef/volumes" Jul 14 22:38:03.154403 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:03.154367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" event={"ID":"c3bdb621-4037-4a14-bb35-fb86912e1f05","Type":"ContainerStarted","Data":"eb9e155bb61d8c2d2f42e948e5c24e9df6e4ed521bbfcb219fc505ebcfb6cbfb"} Jul 14 22:38:03.154756 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:03.154425 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" event={"ID":"c3bdb621-4037-4a14-bb35-fb86912e1f05","Type":"ContainerStarted","Data":"3e26890146856ecb07896905df0bd2bf2a46b4346d68afeaa3b29deaf68e7f64"} Jul 14 22:38:03.172538 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:03.172488 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" podStartSLOduration=2.172472455 podStartE2EDuration="2.172472455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:38:03.170163063 +0000 UTC m=+1748.982794717" watchObservedRunningTime="2026-07-14 22:38:03.172472455 +0000 UTC m=+1748.985104111" Jul 14 22:38:04.161651 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:04.161614 2578 generic.go:356] "Generic (PLEG): container finished" podID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerID="eb9e155bb61d8c2d2f42e948e5c24e9df6e4ed521bbfcb219fc505ebcfb6cbfb" exitCode=7 Jul 14 22:38:04.162039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:04.161690 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" event={"ID":"c3bdb621-4037-4a14-bb35-fb86912e1f05","Type":"ContainerDied","Data":"eb9e155bb61d8c2d2f42e948e5c24e9df6e4ed521bbfcb219fc505ebcfb6cbfb"} Jul 14 22:38:04.162039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:04.161850 2578 scope.go:117] "RemoveContainer" containerID="eb9e155bb61d8c2d2f42e948e5c24e9df6e4ed521bbfcb219fc505ebcfb6cbfb" Jul 14 22:38:05.169562 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:05.169523 2578 generic.go:356] "Generic (PLEG): container finished" podID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerID="231e48c719464019a52f789cd4c2cf5d10d953e3e3bd74d38a3954abc12c5afd" exitCode=7 Jul 14 22:38:05.170066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:05.169608 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" event={"ID":"c3bdb621-4037-4a14-bb35-fb86912e1f05","Type":"ContainerDied","Data":"231e48c719464019a52f789cd4c2cf5d10d953e3e3bd74d38a3954abc12c5afd"} Jul 14 22:38:05.170066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:05.169682 2578 scope.go:117] "RemoveContainer" containerID="eb9e155bb61d8c2d2f42e948e5c24e9df6e4ed521bbfcb219fc505ebcfb6cbfb" Jul 14 22:38:05.170066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:05.169909 2578 scope.go:117] "RemoveContainer" containerID="231e48c719464019a52f789cd4c2cf5d10d953e3e3bd74d38a3954abc12c5afd" Jul 14 22:38:05.170279 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:38:05.170175 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr_odh-ai-gateway-infra(c3bdb621-4037-4a14-bb35-fb86912e1f05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" Jul 14 22:38:05.198326 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:05.198294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:38:06.176314 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:06.176291 2578 scope.go:117] "RemoveContainer" containerID="231e48c719464019a52f789cd4c2cf5d10d953e3e3bd74d38a3954abc12c5afd" Jul 14 22:38:06.176776 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:38:06.176474 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr_odh-ai-gateway-infra(c3bdb621-4037-4a14-bb35-fb86912e1f05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" Jul 14 22:38:13.119715 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:13.119679 2578 patch_prober.go:28] interesting pod/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 22:38:13.120118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:13.119740 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 22:38:13.214498 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:13.214468 2578 generic.go:356] "Generic (PLEG): container finished" podID="a51e3584-46a5-4665-b745-b07b985852b3" containerID="6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35" exitCode=0 Jul 14 22:38:13.214657 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:13.214522 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" event={"ID":"a51e3584-46a5-4665-b745-b07b985852b3","Type":"ContainerDied","Data":"6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35"} Jul 14 22:38:13.214881 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:13.214868 2578 scope.go:117] "RemoveContainer" containerID="6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35" Jul 14 22:38:13.215789 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:13.215760 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:38:13.215841 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:13.215826 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:38:13.215879 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:13.215854 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:38:14.220547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:14.220515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" event={"ID":"a51e3584-46a5-4665-b745-b07b985852b3","Type":"ContainerStarted","Data":"207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba"} Jul 14 22:38:14.469334 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:14.469289 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:38:14.469557 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:14.469533 2578 patch_prober.go:28] interesting pod/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 22:38:14.469614 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:14.469594 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 22:38:15.470223 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:15.470182 2578 patch_prober.go:28] interesting pod/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 22:38:15.470654 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:15.470275 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 22:38:16.469850 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:16.469815 2578 patch_prober.go:28] interesting pod/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 22:38:16.470032 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:16.469872 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 22:38:17.469155 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:17.469118 2578 patch_prober.go:28] interesting pod/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 22:38:17.469580 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:17.469186 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 22:38:18.473624 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:18.473593 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:38:18.473995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:18.473978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:38:18.474623 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:18.474606 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:38:20.831515 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:20.831488 2578 scope.go:117] "RemoveContainer" containerID="231e48c719464019a52f789cd4c2cf5d10d953e3e3bd74d38a3954abc12c5afd" Jul 14 22:38:21.257055 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:21.257022 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" event={"ID":"c3bdb621-4037-4a14-bb35-fb86912e1f05","Type":"ContainerStarted","Data":"b4a9bb711a70db26d33e386c778fb33a12091a21532700aca66ec3af9079f451"} Jul 14 22:38:22.263741 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:22.263706 2578 generic.go:356] "Generic (PLEG): container finished" podID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerID="b4a9bb711a70db26d33e386c778fb33a12091a21532700aca66ec3af9079f451" exitCode=0 Jul 14 22:38:22.264118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:22.263774 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" event={"ID":"c3bdb621-4037-4a14-bb35-fb86912e1f05","Type":"ContainerDied","Data":"b4a9bb711a70db26d33e386c778fb33a12091a21532700aca66ec3af9079f451"} Jul 14 22:38:22.264118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:22.263836 2578 scope.go:117] "RemoveContainer" containerID="231e48c719464019a52f789cd4c2cf5d10d953e3e3bd74d38a3954abc12c5afd" Jul 14 22:38:22.290698 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:22.290672 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr"] Jul 14 22:38:23.407175 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:23.407151 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" Jul 14 22:38:23.497879 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:23.497854 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdb621-4037-4a14-bb35-fb86912e1f05-maas-api-service-ca\") pod \"c3bdb621-4037-4a14-bb35-fb86912e1f05\" (UID: \"c3bdb621-4037-4a14-bb35-fb86912e1f05\") " Jul 14 22:38:23.498165 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:23.498145 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3bdb621-4037-4a14-bb35-fb86912e1f05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3bdb621-4037-4a14-bb35-fb86912e1f05" (UID: "c3bdb621-4037-4a14-bb35-fb86912e1f05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:38:23.599136 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:23.599111 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdb621-4037-4a14-bb35-fb86912e1f05-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:38:24.275781 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:24.275752 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" event={"ID":"c3bdb621-4037-4a14-bb35-fb86912e1f05","Type":"ContainerDied","Data":"3e26890146856ecb07896905df0bd2bf2a46b4346d68afeaa3b29deaf68e7f64"} Jul 14 22:38:24.275947 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:24.275786 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr" Jul 14 22:38:24.275947 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:24.275797 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e26890146856ecb07896905df0bd2bf2a46b4346d68afeaa3b29deaf68e7f64" Jul 14 22:38:24.307441 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:24.307416 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr"] Jul 14 22:38:24.313826 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:24.313803 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-8kbjr"] Jul 14 22:38:24.837311 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:24.837280 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" path="/var/lib/kubelet/pods/c3bdb621-4037-4a14-bb35-fb86912e1f05/volumes" Jul 14 22:38:54.774931 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:54.774903 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:38:54.777939 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:38:54.777914 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:39:02.030397 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.030355 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82"] Jul 14 22:39:02.030923 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.030904 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerName="revoke-keys" Jul 14 22:39:02.031007 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.030927 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerName="revoke-keys" Jul 14 22:39:02.031007 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.030947 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerName="revoke-keys" Jul 14 22:39:02.031007 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.030955 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerName="revoke-keys" Jul 14 22:39:02.031156 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.031103 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerName="revoke-keys" Jul 14 22:39:02.031156 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.031115 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerName="revoke-keys" Jul 14 22:39:02.034222 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.034202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" Jul 14 22:39:02.037057 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.037031 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:39:02.037177 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.037043 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:39:02.040018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.039989 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284f1df-9800-4568-9570-8aeab9f289a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91e25165-9zs82\" (UID: \"f284f1df-9800-4568-9570-8aeab9f289a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" Jul 14 22:39:02.041184 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.041161 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82"] Jul 14 22:39:02.099532 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.099495 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p"] Jul 14 22:39:02.099753 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.099727 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" containerID="cri-o://207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba" gracePeriod=30 Jul 14 22:39:02.141257 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.141210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284f1df-9800-4568-9570-8aeab9f289a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91e25165-9zs82\" (UID: \"f284f1df-9800-4568-9570-8aeab9f289a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" Jul 14 22:39:02.141819 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.141799 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284f1df-9800-4568-9570-8aeab9f289a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91e25165-9zs82\" (UID: \"f284f1df-9800-4568-9570-8aeab9f289a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" Jul 14 22:39:02.346986 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.346945 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" Jul 14 22:39:02.497809 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:02.497777 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82"] Jul 14 22:39:02.499117 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:39:02.499069 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf284f1df_9800_4568_9570_8aeab9f289a7.slice/crio-0a12ed030ecca3a960c58d1b9cc5f4ade7e98c0c01e5cfd7d3c65bb81c3d94d3 WatchSource:0}: Error finding container 0a12ed030ecca3a960c58d1b9cc5f4ade7e98c0c01e5cfd7d3c65bb81c3d94d3: Status 404 returned error can't find the container with id 0a12ed030ecca3a960c58d1b9cc5f4ade7e98c0c01e5cfd7d3c65bb81c3d94d3 Jul 14 22:39:03.476042 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:03.475999 2578 patch_prober.go:28] interesting pod/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:39:03.476539 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:03.476070 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:39:03.482067 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:03.482037 2578 generic.go:356] "Generic (PLEG): container finished" podID="f284f1df-9800-4568-9570-8aeab9f289a7" containerID="596afe194ac169a0d4f5efee53ff4d30a3b7de9103e4c76b08ce743c510e3ac7" exitCode=0 Jul 14 22:39:03.482229 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:03.482122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" event={"ID":"f284f1df-9800-4568-9570-8aeab9f289a7","Type":"ContainerDied","Data":"596afe194ac169a0d4f5efee53ff4d30a3b7de9103e4c76b08ce743c510e3ac7"} Jul 14 22:39:03.482229 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:03.482161 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" event={"ID":"f284f1df-9800-4568-9570-8aeab9f289a7","Type":"ContainerStarted","Data":"0a12ed030ecca3a960c58d1b9cc5f4ade7e98c0c01e5cfd7d3c65bb81c3d94d3"} Jul 14 22:39:04.618424 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:04.618398 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" Jul 14 22:39:04.661731 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:04.661707 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284f1df-9800-4568-9570-8aeab9f289a7-maas-api-service-ca\") pod \"f284f1df-9800-4568-9570-8aeab9f289a7\" (UID: \"f284f1df-9800-4568-9570-8aeab9f289a7\") " Jul 14 22:39:04.662036 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:04.662014 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f284f1df-9800-4568-9570-8aeab9f289a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f284f1df-9800-4568-9570-8aeab9f289a7" (UID: "f284f1df-9800-4568-9570-8aeab9f289a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:39:04.763155 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:04.763073 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f284f1df-9800-4568-9570-8aeab9f289a7-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:05.094661 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:05.094621 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:39:05.493783 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:05.493689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" event={"ID":"f284f1df-9800-4568-9570-8aeab9f289a7","Type":"ContainerDied","Data":"0a12ed030ecca3a960c58d1b9cc5f4ade7e98c0c01e5cfd7d3c65bb81c3d94d3"} Jul 14 22:39:05.493783 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:05.493727 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91e25165-9zs82" Jul 14 22:39:05.493783 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:05.493751 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a12ed030ecca3a960c58d1b9cc5f4ade7e98c0c01e5cfd7d3c65bb81c3d94d3" Jul 14 22:39:07.364567 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.364511 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:39:07.387353 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387329 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-data\") pod \"a51e3584-46a5-4665-b745-b07b985852b3\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " Jul 14 22:39:07.387466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387379 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51e3584-46a5-4665-b745-b07b985852b3-istiod-ca-cert\") pod \"a51e3584-46a5-4665-b745-b07b985852b3\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " Jul 14 22:39:07.387466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387396 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51e3584-46a5-4665-b745-b07b985852b3-istio-podinfo\") pod \"a51e3584-46a5-4665-b745-b07b985852b3\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " Jul 14 22:39:07.387466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387425 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-credential-socket\") pod \"a51e3584-46a5-4665-b745-b07b985852b3\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " Jul 14 22:39:07.387466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387447 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvlz4\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-kube-api-access-cvlz4\") pod \"a51e3584-46a5-4665-b745-b07b985852b3\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " Jul 14 22:39:07.387732 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387477 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-certs\") pod \"a51e3584-46a5-4665-b745-b07b985852b3\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " Jul 14 22:39:07.387732 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387504 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-envoy\") pod \"a51e3584-46a5-4665-b745-b07b985852b3\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " Jul 14 22:39:07.387732 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387536 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-socket\") pod \"a51e3584-46a5-4665-b745-b07b985852b3\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " Jul 14 22:39:07.387890 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387737 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a51e3584-46a5-4665-b745-b07b985852b3" (UID: "a51e3584-46a5-4665-b745-b07b985852b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:39:07.387890 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387769 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a51e3584-46a5-4665-b745-b07b985852b3" (UID: "a51e3584-46a5-4665-b745-b07b985852b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:39:07.387890 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387781 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a51e3584-46a5-4665-b745-b07b985852b3" (UID: "a51e3584-46a5-4665-b745-b07b985852b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:39:07.387890 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387792 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-istio-token\") pod \"a51e3584-46a5-4665-b745-b07b985852b3\" (UID: \"a51e3584-46a5-4665-b745-b07b985852b3\") " Jul 14 22:39:07.387890 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.387808 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51e3584-46a5-4665-b745-b07b985852b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a51e3584-46a5-4665-b745-b07b985852b3" (UID: "a51e3584-46a5-4665-b745-b07b985852b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:39:07.388161 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.388143 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:07.388210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.388164 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51e3584-46a5-4665-b745-b07b985852b3-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:07.388210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.388179 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:07.388210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.388195 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:07.388541 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.388517 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "a51e3584-46a5-4665-b745-b07b985852b3" (UID: "a51e3584-46a5-4665-b745-b07b985852b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:39:07.389755 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.389729 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a51e3584-46a5-4665-b745-b07b985852b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a51e3584-46a5-4665-b745-b07b985852b3" (UID: "a51e3584-46a5-4665-b745-b07b985852b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:39:07.389846 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.389783 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a51e3584-46a5-4665-b745-b07b985852b3" (UID: "a51e3584-46a5-4665-b745-b07b985852b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:39:07.390036 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.390019 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-kube-api-access-cvlz4" (OuterVolumeSpecName: "kube-api-access-cvlz4") pod "a51e3584-46a5-4665-b745-b07b985852b3" (UID: "a51e3584-46a5-4665-b745-b07b985852b3"). InnerVolumeSpecName "kube-api-access-cvlz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:39:07.390095 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.390033 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "a51e3584-46a5-4665-b745-b07b985852b3" (UID: "a51e3584-46a5-4665-b745-b07b985852b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:39:07.489327 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.489299 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:07.489327 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.489322 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51e3584-46a5-4665-b745-b07b985852b3-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:07.489494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.489332 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvlz4\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-kube-api-access-cvlz4\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:07.489494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.489342 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51e3584-46a5-4665-b745-b07b985852b3-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:07.489494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.489350 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51e3584-46a5-4665-b745-b07b985852b3-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:07.508291 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.508260 2578 generic.go:356] "Generic (PLEG): container finished" podID="a51e3584-46a5-4665-b745-b07b985852b3" containerID="207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba" exitCode=0 Jul 14 22:39:07.508448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.508306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" event={"ID":"a51e3584-46a5-4665-b745-b07b985852b3","Type":"ContainerDied","Data":"207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba"} Jul 14 22:39:07.508448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.508329 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" Jul 14 22:39:07.508448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.508350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p" event={"ID":"a51e3584-46a5-4665-b745-b07b985852b3","Type":"ContainerDied","Data":"430b99e08bd9cea8363fe95c1d391955aabc5349444fccafd8bb127f8aa7d949"} Jul 14 22:39:07.508448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.508372 2578 scope.go:117] "RemoveContainer" containerID="207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba" Jul 14 22:39:07.517958 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.517940 2578 scope.go:117] "RemoveContainer" containerID="6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35" Jul 14 22:39:07.525938 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.525917 2578 scope.go:117] "RemoveContainer" containerID="207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba" Jul 14 22:39:07.526169 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:39:07.526154 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba\": container with ID starting with 207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba not found: ID does not exist" containerID="207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba" Jul 14 22:39:07.526210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.526176 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba"} err="failed to get container status \"207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba\": rpc error: code = NotFound desc = could not find container \"207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba\": container with ID starting with 207ff786af08e7f1ee81cb9af932bfa4327f81fa904a7bc1ab3a1cb1540fffba not found: ID does not exist" Jul 14 22:39:07.526210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.526192 2578 scope.go:117] "RemoveContainer" containerID="6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35" Jul 14 22:39:07.526423 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:39:07.526410 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35\": container with ID starting with 6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35 not found: ID does not exist" containerID="6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35" Jul 14 22:39:07.526473 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.526429 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35"} err="failed to get container status \"6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35\": rpc error: code = NotFound desc = could not find container \"6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35\": container with ID starting with 6ba9dbc0dc9d73d57910806ceefe4e56f5ada6440bbd8175206d647c3b2b8e35 not found: ID does not exist" Jul 14 22:39:07.532991 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.532971 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p"] Jul 14 22:39:07.540412 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:07.540389 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-91e25165-gw-openshift-default-5d8875c4c-h6r4p"] Jul 14 22:39:08.837740 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:08.837701 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51e3584-46a5-4665-b745-b07b985852b3" path="/var/lib/kubelet/pods/a51e3584-46a5-4665-b745-b07b985852b3/volumes" Jul 14 22:39:14.855654 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.855624 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw"] Jul 14 22:39:14.856066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.855974 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" Jul 14 22:39:14.856066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.855984 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" Jul 14 22:39:14.856066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.855994 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" Jul 14 22:39:14.856066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.855999 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" Jul 14 22:39:14.856066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.856008 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f284f1df-9800-4568-9570-8aeab9f289a7" containerName="revoke-keys" Jul 14 22:39:14.856066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.856014 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f284f1df-9800-4568-9570-8aeab9f289a7" containerName="revoke-keys" Jul 14 22:39:14.856066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.856024 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerName="revoke-keys" Jul 14 22:39:14.856066 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.856029 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerName="revoke-keys" Jul 14 22:39:14.856525 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.856079 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f284f1df-9800-4568-9570-8aeab9f289a7" containerName="revoke-keys" Jul 14 22:39:14.856525 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.856088 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3bdb621-4037-4a14-bb35-fb86912e1f05" containerName="revoke-keys" Jul 14 22:39:14.856525 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.856099 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" Jul 14 22:39:14.856525 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.856106 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51e3584-46a5-4665-b745-b07b985852b3" containerName="istio-proxy" Jul 14 22:39:14.860582 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.860561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:14.863332 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.863314 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-577df277-openshift-default-dockercfg-jgtjh\"" Jul 14 22:39:14.877407 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.877385 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw"] Jul 14 22:39:14.955040 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.955008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-socket\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:14.955223 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.955050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-podinfo\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:14.955223 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.955107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7dsl\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-kube-api-access-l7dsl\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:14.955223 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.955168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-certs\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:14.955223 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.955208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-credential-socket\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:14.955430 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.955229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istiod-ca-cert\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:14.955430 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.955381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-data\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:14.955566 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.955504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-envoy\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:14.955633 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:14.955583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-token\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057316 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dsl\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-kube-api-access-l7dsl\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-certs\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-credential-socket\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istiod-ca-cert\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-data\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-envoy\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057494 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-token\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057792 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-socket\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057852 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-podinfo\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057919 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-certs\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.057975 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.057919 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-credential-socket\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.058032 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.058011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-data\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.058089 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.058059 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-socket\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.058176 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.058156 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istiod-ca-cert\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.060137 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.060114 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-envoy\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.060302 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.060287 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-podinfo\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.066698 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.066678 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-token\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.067989 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.067961 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7dsl\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-kube-api-access-l7dsl\") pod \"e2e-ait-577df277-openshift-default-78b765b47c-cwlxw\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.172020 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.171935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:15.322673 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.322649 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw"] Jul 14 22:39:15.324821 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:39:15.324793 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ea73b0_3dbb_4ce3_af6a_ea58d550e361.slice/crio-db7aeb4928af759bdee211d296b37e9139115fce2a602a405091e0a36b75a084 WatchSource:0}: Error finding container db7aeb4928af759bdee211d296b37e9139115fce2a602a405091e0a36b75a084: Status 404 returned error can't find the container with id db7aeb4928af759bdee211d296b37e9139115fce2a602a405091e0a36b75a084 Jul 14 22:39:15.326863 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.326831 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:39:15.326947 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.326897 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:39:15.326947 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.326929 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:39:15.550629 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.550538 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" event={"ID":"93ea73b0-3dbb-4ce3-af6a-ea58d550e361","Type":"ContainerStarted","Data":"c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227"} Jul 14 22:39:15.550629 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.550596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" event={"ID":"93ea73b0-3dbb-4ce3-af6a-ea58d550e361","Type":"ContainerStarted","Data":"db7aeb4928af759bdee211d296b37e9139115fce2a602a405091e0a36b75a084"} Jul 14 22:39:15.578030 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:15.577959 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" podStartSLOduration=1.57793652 podStartE2EDuration="1.57793652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:39:15.577339554 +0000 UTC m=+1821.389971195" watchObservedRunningTime="2026-07-14 22:39:15.57793652 +0000 UTC m=+1821.390568160" Jul 14 22:39:16.173056 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:16.173024 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:16.177786 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:16.177761 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:16.555940 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:16.555917 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:16.556996 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:16.556975 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:21.177124 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.177090 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj"] Jul 14 22:39:21.180793 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.180775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" Jul 14 22:39:21.184065 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.184040 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:39:21.184192 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.184081 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:39:21.190936 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.190912 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj"] Jul 14 22:39:21.316652 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.316618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e30180-f037-4d1e-aa05-e6e527cca311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-577df277-p5wcj\" (UID: \"87e30180-f037-4d1e-aa05-e6e527cca311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" Jul 14 22:39:21.418091 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.418050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e30180-f037-4d1e-aa05-e6e527cca311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-577df277-p5wcj\" (UID: \"87e30180-f037-4d1e-aa05-e6e527cca311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" Jul 14 22:39:21.418757 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.418735 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e30180-f037-4d1e-aa05-e6e527cca311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-577df277-p5wcj\" (UID: \"87e30180-f037-4d1e-aa05-e6e527cca311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" Jul 14 22:39:21.493055 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.492982 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" Jul 14 22:39:21.642470 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:21.642446 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj"] Jul 14 22:39:21.644703 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:39:21.644679 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e30180_f037_4d1e_aa05_e6e527cca311.slice/crio-b60d59caafad19f9d810f81549a92541a63ce8bd8cfed8384c5db927ee135d56 WatchSource:0}: Error finding container b60d59caafad19f9d810f81549a92541a63ce8bd8cfed8384c5db927ee135d56: Status 404 returned error can't find the container with id b60d59caafad19f9d810f81549a92541a63ce8bd8cfed8384c5db927ee135d56 Jul 14 22:39:22.588704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:22.588669 2578 generic.go:356] "Generic (PLEG): container finished" podID="87e30180-f037-4d1e-aa05-e6e527cca311" containerID="ccc80e6d43e64451ebd7bcdcdeac0ae4df36f64f7f84a8689fc48ce9a14ff115" exitCode=7 Jul 14 22:39:22.589094 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:22.588751 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" event={"ID":"87e30180-f037-4d1e-aa05-e6e527cca311","Type":"ContainerDied","Data":"ccc80e6d43e64451ebd7bcdcdeac0ae4df36f64f7f84a8689fc48ce9a14ff115"} Jul 14 22:39:22.589094 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:22.588810 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" event={"ID":"87e30180-f037-4d1e-aa05-e6e527cca311","Type":"ContainerStarted","Data":"b60d59caafad19f9d810f81549a92541a63ce8bd8cfed8384c5db927ee135d56"} Jul 14 22:39:22.589094 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:22.588939 2578 scope.go:117] "RemoveContainer" containerID="ccc80e6d43e64451ebd7bcdcdeac0ae4df36f64f7f84a8689fc48ce9a14ff115" Jul 14 22:39:23.494684 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:23.494596 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:39:23.595537 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:23.595499 2578 generic.go:356] "Generic (PLEG): container finished" podID="87e30180-f037-4d1e-aa05-e6e527cca311" containerID="b82f1ed96e69f66013890f210008909d48a22017894382854a348ffdd3d5fa59" exitCode=7 Jul 14 22:39:23.596017 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:23.595601 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" event={"ID":"87e30180-f037-4d1e-aa05-e6e527cca311","Type":"ContainerDied","Data":"b82f1ed96e69f66013890f210008909d48a22017894382854a348ffdd3d5fa59"} Jul 14 22:39:23.596017 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:23.595662 2578 scope.go:117] "RemoveContainer" containerID="ccc80e6d43e64451ebd7bcdcdeac0ae4df36f64f7f84a8689fc48ce9a14ff115" Jul 14 22:39:23.596017 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:23.595755 2578 scope.go:117] "RemoveContainer" containerID="b82f1ed96e69f66013890f210008909d48a22017894382854a348ffdd3d5fa59" Jul 14 22:39:23.596017 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:39:23.595966 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-577df277-p5wcj_odh-ai-gateway-infra(87e30180-f037-4d1e-aa05-e6e527cca311)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" Jul 14 22:39:24.603581 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:24.603540 2578 scope.go:117] "RemoveContainer" containerID="b82f1ed96e69f66013890f210008909d48a22017894382854a348ffdd3d5fa59" Jul 14 22:39:24.604038 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:39:24.603734 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-577df277-p5wcj_odh-ai-gateway-infra(87e30180-f037-4d1e-aa05-e6e527cca311)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" Jul 14 22:39:31.578786 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:31.577863 2578 patch_prober.go:28] interesting pod/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": read tcp 10.132.0.2:43698->10.132.0.52:15021: read: connection reset by peer" start-of-body= Jul 14 22:39:31.578786 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:31.577947 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": read tcp 10.132.0.2:43698->10.132.0.52:15021: read: connection reset by peer" Jul 14 22:39:31.639392 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:31.639350 2578 generic.go:356] "Generic (PLEG): container finished" podID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerID="c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227" exitCode=0 Jul 14 22:39:31.639559 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:31.639421 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" event={"ID":"93ea73b0-3dbb-4ce3-af6a-ea58d550e361","Type":"ContainerDied","Data":"c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227"} Jul 14 22:39:31.639855 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:31.639841 2578 scope.go:117] "RemoveContainer" containerID="c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227" Jul 14 22:39:31.640859 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:31.640827 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:39:31.640915 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:31.640902 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:39:31.640956 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:31.640944 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:39:32.651091 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:32.651034 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" event={"ID":"93ea73b0-3dbb-4ce3-af6a-ea58d550e361","Type":"ContainerStarted","Data":"f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3"} Jul 14 22:39:33.173096 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:33.173061 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:33.173350 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:33.173326 2578 patch_prober.go:28] interesting pod/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 22:39:33.173425 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:33.173376 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 22:39:34.172596 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:34.172561 2578 patch_prober.go:28] interesting pod/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 22:39:34.172958 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:34.172618 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 22:39:35.172341 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:35.172304 2578 patch_prober.go:28] interesting pod/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 22:39:35.172543 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:35.172356 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 22:39:35.832207 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:35.832172 2578 scope.go:117] "RemoveContainer" containerID="b82f1ed96e69f66013890f210008909d48a22017894382854a348ffdd3d5fa59" Jul 14 22:39:36.172576 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:36.172545 2578 patch_prober.go:28] interesting pod/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 22:39:36.172681 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:36.172586 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 22:39:36.677116 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:36.677084 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" event={"ID":"87e30180-f037-4d1e-aa05-e6e527cca311","Type":"ContainerStarted","Data":"0397dadcf16ad54fe7624a76dfea4a117a0d92c15416b2b3b0da07c8ea0274b8"} Jul 14 22:39:36.702323 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:36.702275 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" podStartSLOduration=15.702259971 podStartE2EDuration="15.702259971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:39:36.700062313 +0000 UTC m=+1842.512693953" watchObservedRunningTime="2026-07-14 22:39:36.702259971 +0000 UTC m=+1842.514891605" Jul 14 22:39:37.176389 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:37.176356 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:37.176789 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:37.176739 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:37.177369 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:37.177348 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:39:37.684283 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:37.684224 2578 generic.go:356] "Generic (PLEG): container finished" podID="87e30180-f037-4d1e-aa05-e6e527cca311" containerID="0397dadcf16ad54fe7624a76dfea4a117a0d92c15416b2b3b0da07c8ea0274b8" exitCode=0 Jul 14 22:39:37.684283 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:37.684282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" event={"ID":"87e30180-f037-4d1e-aa05-e6e527cca311","Type":"ContainerDied","Data":"0397dadcf16ad54fe7624a76dfea4a117a0d92c15416b2b3b0da07c8ea0274b8"} Jul 14 22:39:37.684508 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:37.684337 2578 scope.go:117] "RemoveContainer" containerID="b82f1ed96e69f66013890f210008909d48a22017894382854a348ffdd3d5fa59" Jul 14 22:39:37.720390 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:37.720365 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj"] Jul 14 22:39:38.827606 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:38.827587 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" Jul 14 22:39:38.993929 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:38.993840 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e30180-f037-4d1e-aa05-e6e527cca311-maas-api-service-ca\") pod \"87e30180-f037-4d1e-aa05-e6e527cca311\" (UID: \"87e30180-f037-4d1e-aa05-e6e527cca311\") " Jul 14 22:39:38.994260 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:38.994218 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e30180-f037-4d1e-aa05-e6e527cca311-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87e30180-f037-4d1e-aa05-e6e527cca311" (UID: "87e30180-f037-4d1e-aa05-e6e527cca311"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:39:39.095122 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:39.095080 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e30180-f037-4d1e-aa05-e6e527cca311-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:39:39.697001 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:39.696973 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" Jul 14 22:39:39.697201 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:39.697016 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj" event={"ID":"87e30180-f037-4d1e-aa05-e6e527cca311","Type":"ContainerDied","Data":"b60d59caafad19f9d810f81549a92541a63ce8bd8cfed8384c5db927ee135d56"} Jul 14 22:39:39.697201 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:39.697067 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b60d59caafad19f9d810f81549a92541a63ce8bd8cfed8384c5db927ee135d56" Jul 14 22:39:39.728249 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:39.728207 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj"] Jul 14 22:39:39.731429 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:39.731405 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p5wcj"] Jul 14 22:39:40.836966 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:39:40.836933 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" path="/var/lib/kubelet/pods/87e30180-f037-4d1e-aa05-e6e527cca311/volumes" Jul 14 22:40:11.288361 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.288329 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh"] Jul 14 22:40:11.288803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.288704 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" containerName="revoke-keys" Jul 14 22:40:11.288803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.288715 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" containerName="revoke-keys" Jul 14 22:40:11.288803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.288730 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" containerName="revoke-keys" Jul 14 22:40:11.288803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.288735 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" containerName="revoke-keys" Jul 14 22:40:11.288803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.288802 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" containerName="revoke-keys" Jul 14 22:40:11.288990 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.288810 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" containerName="revoke-keys" Jul 14 22:40:11.291945 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.291928 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" Jul 14 22:40:11.297290 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.297259 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:40:11.297290 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.297259 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:40:11.308649 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.308624 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh"] Jul 14 22:40:11.367639 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.367604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8131e82-2f07-4818-bdb3-1a14d832e6b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-577df277-p2bbh\" (UID: \"e8131e82-2f07-4818-bdb3-1a14d832e6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" Jul 14 22:40:11.468618 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.468583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8131e82-2f07-4818-bdb3-1a14d832e6b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-577df277-p2bbh\" (UID: \"e8131e82-2f07-4818-bdb3-1a14d832e6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" Jul 14 22:40:11.469190 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.469172 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8131e82-2f07-4818-bdb3-1a14d832e6b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-577df277-p2bbh\" (UID: \"e8131e82-2f07-4818-bdb3-1a14d832e6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" Jul 14 22:40:11.601707 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.601674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" Jul 14 22:40:11.760269 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.760231 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh"] Jul 14 22:40:11.762472 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:40:11.762443 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8131e82_2f07_4818_bdb3_1a14d832e6b3.slice/crio-b52b9490a66b73dbc0016a5b4f513affc809e113ddfb5091b8fb71b349119057 WatchSource:0}: Error finding container b52b9490a66b73dbc0016a5b4f513affc809e113ddfb5091b8fb71b349119057: Status 404 returned error can't find the container with id b52b9490a66b73dbc0016a5b4f513affc809e113ddfb5091b8fb71b349119057 Jul 14 22:40:11.860805 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:11.860720 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" event={"ID":"e8131e82-2f07-4818-bdb3-1a14d832e6b3","Type":"ContainerStarted","Data":"b52b9490a66b73dbc0016a5b4f513affc809e113ddfb5091b8fb71b349119057"} Jul 14 22:40:12.867472 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:12.867441 2578 generic.go:356] "Generic (PLEG): container finished" podID="e8131e82-2f07-4818-bdb3-1a14d832e6b3" containerID="60546676a10a16cb17e5bcbd7b485aa27c27516ca312e63cfd02ed38f6300654" exitCode=0 Jul 14 22:40:12.867854 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:12.867483 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" event={"ID":"e8131e82-2f07-4818-bdb3-1a14d832e6b3","Type":"ContainerDied","Data":"60546676a10a16cb17e5bcbd7b485aa27c27516ca312e63cfd02ed38f6300654"} Jul 14 22:40:14.010288 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:14.010261 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" Jul 14 22:40:14.090365 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:14.090334 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8131e82-2f07-4818-bdb3-1a14d832e6b3-maas-api-service-ca\") pod \"e8131e82-2f07-4818-bdb3-1a14d832e6b3\" (UID: \"e8131e82-2f07-4818-bdb3-1a14d832e6b3\") " Jul 14 22:40:14.090664 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:14.090643 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8131e82-2f07-4818-bdb3-1a14d832e6b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8131e82-2f07-4818-bdb3-1a14d832e6b3" (UID: "e8131e82-2f07-4818-bdb3-1a14d832e6b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:40:14.190999 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:14.190909 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8131e82-2f07-4818-bdb3-1a14d832e6b3-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:14.877906 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:14.877880 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" Jul 14 22:40:14.878037 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:14.877918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-577df277-p2bbh" event={"ID":"e8131e82-2f07-4818-bdb3-1a14d832e6b3","Type":"ContainerDied","Data":"b52b9490a66b73dbc0016a5b4f513affc809e113ddfb5091b8fb71b349119057"} Jul 14 22:40:14.878037 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:14.877963 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b52b9490a66b73dbc0016a5b4f513affc809e113ddfb5091b8fb71b349119057" Jul 14 22:40:25.343962 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:25.343868 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:40:37.620894 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.620860 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw"] Jul 14 22:40:37.621354 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.621059 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" containerID="cri-o://f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3" gracePeriod=30 Jul 14 22:40:37.973654 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.973625 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2"] Jul 14 22:40:37.974022 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.974010 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" containerName="revoke-keys" Jul 14 22:40:37.974067 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.974024 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" containerName="revoke-keys" Jul 14 22:40:37.974067 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.974045 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8131e82-2f07-4818-bdb3-1a14d832e6b3" containerName="revoke-keys" Jul 14 22:40:37.974067 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.974051 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8131e82-2f07-4818-bdb3-1a14d832e6b3" containerName="revoke-keys" Jul 14 22:40:37.974202 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.974112 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8131e82-2f07-4818-bdb3-1a14d832e6b3" containerName="revoke-keys" Jul 14 22:40:37.974202 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.974123 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e30180-f037-4d1e-aa05-e6e527cca311" containerName="revoke-keys" Jul 14 22:40:37.977145 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.977128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:37.983690 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.983669 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6205e423-openshift-default-dockercfg-gsw2p\"" Jul 14 22:40:37.995215 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.995189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-token\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:37.995367 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.995221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc748\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-kube-api-access-hc748\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:37.995367 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.995292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-credential-socket\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:37.995367 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.995336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ff7a-9669-4754-a03a-b28dbde7732c-istiod-ca-cert\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:37.995367 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.995355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-envoy\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:37.995545 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.995419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-data\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:37.995545 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.995438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-podinfo\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:37.995545 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.995480 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-socket\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:37.995545 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:37.995522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-certs\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.009348 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.009322 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2"] Jul 14 22:40:38.098254 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.098198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-data\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.098669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.098283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-podinfo\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.098669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.098321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-socket\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.098669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.098357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-certs\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.098669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.098455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-token\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.098669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.098485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc748\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-kube-api-access-hc748\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.098669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.098523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-credential-socket\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.098669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.098603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ff7a-9669-4754-a03a-b28dbde7732c-istiod-ca-cert\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.098669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.098635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-envoy\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.099910 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.099460 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-certs\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.100717 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.100686 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-socket\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.100717 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.100710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-data\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.100999 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.100976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-credential-socket\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.101365 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.101330 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ff7a-9669-4754-a03a-b28dbde7732c-istiod-ca-cert\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.102947 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.102918 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-envoy\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.103442 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.103425 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-podinfo\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.123827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.123804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-token\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.126109 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.126086 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc748\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-kube-api-access-hc748\") pod \"e2e-derive-6205e423-openshift-default-7d45db966c-srgh2\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.287806 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.287718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:38.469339 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.469308 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2"] Jul 14 22:40:38.470794 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:40:38.470771 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc02ff7a_9669_4754_a03a_b28dbde7732c.slice/crio-869cce3a8b0bc564b6bd03673a7fa4d7295c2db313c5bce553382c519971b71a WatchSource:0}: Error finding container 869cce3a8b0bc564b6bd03673a7fa4d7295c2db313c5bce553382c519971b71a: Status 404 returned error can't find the container with id 869cce3a8b0bc564b6bd03673a7fa4d7295c2db313c5bce553382c519971b71a Jul 14 22:40:38.472767 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.472733 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:40:38.472838 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.472798 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:40:38.472877 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:38.472842 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:40:39.002163 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:39.002122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" event={"ID":"cc02ff7a-9669-4754-a03a-b28dbde7732c","Type":"ContainerStarted","Data":"a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7"} Jul 14 22:40:39.002662 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:39.002176 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" event={"ID":"cc02ff7a-9669-4754-a03a-b28dbde7732c","Type":"ContainerStarted","Data":"869cce3a8b0bc564b6bd03673a7fa4d7295c2db313c5bce553382c519971b71a"} Jul 14 22:40:39.033621 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:39.033570 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" podStartSLOduration=2.033556682 podStartE2EDuration="2.033556682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:40:39.033131124 +0000 UTC m=+1904.845762763" watchObservedRunningTime="2026-07-14 22:40:39.033556682 +0000 UTC m=+1904.846188324" Jul 14 22:40:39.288926 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:39.288831 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:39.293736 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:39.293711 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:40.007310 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:40.007278 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:40.008250 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:40.008225 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:42.875362 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.875337 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:40:42.946952 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.946879 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-socket\") pod \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " Jul 14 22:40:42.947110 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.946974 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-envoy\") pod \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " Jul 14 22:40:42.947110 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947003 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-credential-socket\") pod \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " Jul 14 22:40:42.947110 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947040 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7dsl\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-kube-api-access-l7dsl\") pod \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " Jul 14 22:40:42.947110 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947060 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-certs\") pod \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " Jul 14 22:40:42.947110 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947081 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-data\") pod \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " Jul 14 22:40:42.947428 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947118 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-podinfo\") pod \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " Jul 14 22:40:42.947428 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947206 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "93ea73b0-3dbb-4ce3-af6a-ea58d550e361" (UID: "93ea73b0-3dbb-4ce3-af6a-ea58d550e361"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:40:42.947428 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947330 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istiod-ca-cert\") pod \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " Jul 14 22:40:42.947428 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947336 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "93ea73b0-3dbb-4ce3-af6a-ea58d550e361" (UID: "93ea73b0-3dbb-4ce3-af6a-ea58d550e361"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:40:42.947428 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947364 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-token\") pod \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\" (UID: \"93ea73b0-3dbb-4ce3-af6a-ea58d550e361\") " Jul 14 22:40:42.947728 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947705 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "93ea73b0-3dbb-4ce3-af6a-ea58d550e361" (UID: "93ea73b0-3dbb-4ce3-af6a-ea58d550e361"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:40:42.948318 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947865 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "93ea73b0-3dbb-4ce3-af6a-ea58d550e361" (UID: "93ea73b0-3dbb-4ce3-af6a-ea58d550e361"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:40:42.948318 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.947919 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:42.948318 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.948031 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:42.948318 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.948051 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-data" (OuterVolumeSpecName: "istio-data") pod "93ea73b0-3dbb-4ce3-af6a-ea58d550e361" (UID: "93ea73b0-3dbb-4ce3-af6a-ea58d550e361"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:40:42.949813 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.949786 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "93ea73b0-3dbb-4ce3-af6a-ea58d550e361" (UID: "93ea73b0-3dbb-4ce3-af6a-ea58d550e361"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:40:42.949955 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.949918 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-token" (OuterVolumeSpecName: "istio-token") pod "93ea73b0-3dbb-4ce3-af6a-ea58d550e361" (UID: "93ea73b0-3dbb-4ce3-af6a-ea58d550e361"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:40:42.950050 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.950012 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-kube-api-access-l7dsl" (OuterVolumeSpecName: "kube-api-access-l7dsl") pod "93ea73b0-3dbb-4ce3-af6a-ea58d550e361" (UID: "93ea73b0-3dbb-4ce3-af6a-ea58d550e361"). InnerVolumeSpecName "kube-api-access-l7dsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:40:42.950206 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:42.950186 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "93ea73b0-3dbb-4ce3-af6a-ea58d550e361" (UID: "93ea73b0-3dbb-4ce3-af6a-ea58d550e361"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:40:43.023524 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.023495 2578 generic.go:356] "Generic (PLEG): container finished" podID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerID="f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3" exitCode=0 Jul 14 22:40:43.023662 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.023558 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" Jul 14 22:40:43.023662 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.023578 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" event={"ID":"93ea73b0-3dbb-4ce3-af6a-ea58d550e361","Type":"ContainerDied","Data":"f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3"} Jul 14 22:40:43.023662 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.023646 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw" event={"ID":"93ea73b0-3dbb-4ce3-af6a-ea58d550e361","Type":"ContainerDied","Data":"db7aeb4928af759bdee211d296b37e9139115fce2a602a405091e0a36b75a084"} Jul 14 22:40:43.023794 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.023670 2578 scope.go:117] "RemoveContainer" containerID="f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3" Jul 14 22:40:43.033263 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.033198 2578 scope.go:117] "RemoveContainer" containerID="c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227" Jul 14 22:40:43.045216 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.045199 2578 scope.go:117] "RemoveContainer" containerID="f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3" Jul 14 22:40:43.045524 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:40:43.045498 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3\": container with ID starting with f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3 not found: ID does not exist" containerID="f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3" Jul 14 22:40:43.045594 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.045533 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3"} err="failed to get container status \"f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3\": rpc error: code = NotFound desc = could not find container \"f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3\": container with ID starting with f49399291d319b29fec8ee4ef7128aef3d0ce3a1e3eae67611b00a0f13f75df3 not found: ID does not exist" Jul 14 22:40:43.045594 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.045551 2578 scope.go:117] "RemoveContainer" containerID="c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227" Jul 14 22:40:43.045795 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:40:43.045779 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227\": container with ID starting with c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227 not found: ID does not exist" containerID="c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227" Jul 14 22:40:43.045865 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.045801 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227"} err="failed to get container status \"c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227\": rpc error: code = NotFound desc = could not find container \"c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227\": container with ID starting with c22be7d8b3a0f36c150a9c471a69db7dff70831575988e4701ad0356ceddc227 not found: ID does not exist" Jul 14 22:40:43.048644 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.048627 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7dsl\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-kube-api-access-l7dsl\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:43.048716 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.048655 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:43.048716 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.048664 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:43.048716 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.048672 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:43.048716 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.048683 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:43.048716 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.048694 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:43.048716 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.048702 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ea73b0-3dbb-4ce3-af6a-ea58d550e361-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:40:43.075682 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.075660 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw"] Jul 14 22:40:43.091336 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.091287 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-577df277-openshift-default-78b765b47c-cwlxw"] Jul 14 22:40:43.611817 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.611784 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr"] Jul 14 22:40:43.612266 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.612220 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" Jul 14 22:40:43.612266 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.612255 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" Jul 14 22:40:43.612433 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.612400 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" Jul 14 22:40:43.612433 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.612413 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" Jul 14 22:40:43.617067 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.617045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" Jul 14 22:40:43.623970 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.623948 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:40:43.624083 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.623975 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:40:43.633765 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.633742 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr"] Jul 14 22:40:43.655549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.655525 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241f1aff-eddb-44f2-9c9f-6e425b687e47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6205e423-dgjrr\" (UID: \"241f1aff-eddb-44f2-9c9f-6e425b687e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" Jul 14 22:40:43.756427 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.756398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241f1aff-eddb-44f2-9c9f-6e425b687e47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6205e423-dgjrr\" (UID: \"241f1aff-eddb-44f2-9c9f-6e425b687e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" Jul 14 22:40:43.759906 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.757477 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241f1aff-eddb-44f2-9c9f-6e425b687e47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6205e423-dgjrr\" (UID: \"241f1aff-eddb-44f2-9c9f-6e425b687e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" Jul 14 22:40:43.927457 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:43.927377 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" Jul 14 22:40:44.072160 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:44.072128 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr"] Jul 14 22:40:44.073076 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:40:44.073051 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241f1aff_eddb_44f2_9c9f_6e425b687e47.slice/crio-acffedaadc299b45410f04950c896e7608b419df0977b4120e580a4731c1a324 WatchSource:0}: Error finding container acffedaadc299b45410f04950c896e7608b419df0977b4120e580a4731c1a324: Status 404 returned error can't find the container with id acffedaadc299b45410f04950c896e7608b419df0977b4120e580a4731c1a324 Jul 14 22:40:44.837787 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:44.837760 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" path="/var/lib/kubelet/pods/93ea73b0-3dbb-4ce3-af6a-ea58d550e361/volumes" Jul 14 22:40:45.040118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:45.040017 2578 generic.go:356] "Generic (PLEG): container finished" podID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerID="1930c0aa0732081600ad931e79651cc0e901c1b4f88484995bbcd3b958b1016a" exitCode=7 Jul 14 22:40:45.040118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:45.040099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" event={"ID":"241f1aff-eddb-44f2-9c9f-6e425b687e47","Type":"ContainerDied","Data":"1930c0aa0732081600ad931e79651cc0e901c1b4f88484995bbcd3b958b1016a"} Jul 14 22:40:45.040573 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:45.040150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" event={"ID":"241f1aff-eddb-44f2-9c9f-6e425b687e47","Type":"ContainerStarted","Data":"acffedaadc299b45410f04950c896e7608b419df0977b4120e580a4731c1a324"} Jul 14 22:40:45.040573 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:45.040284 2578 scope.go:117] "RemoveContainer" containerID="1930c0aa0732081600ad931e79651cc0e901c1b4f88484995bbcd3b958b1016a" Jul 14 22:40:46.047919 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:46.047824 2578 generic.go:356] "Generic (PLEG): container finished" podID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerID="7e158360c9e2067dcabc562c9b2b1cde35bfa2aa74d11d12cea19dcee3ceebe2" exitCode=7 Jul 14 22:40:46.047919 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:46.047868 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" event={"ID":"241f1aff-eddb-44f2-9c9f-6e425b687e47","Type":"ContainerDied","Data":"7e158360c9e2067dcabc562c9b2b1cde35bfa2aa74d11d12cea19dcee3ceebe2"} Jul 14 22:40:46.047919 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:46.047915 2578 scope.go:117] "RemoveContainer" containerID="1930c0aa0732081600ad931e79651cc0e901c1b4f88484995bbcd3b958b1016a" Jul 14 22:40:46.048508 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:46.048119 2578 scope.go:117] "RemoveContainer" containerID="7e158360c9e2067dcabc562c9b2b1cde35bfa2aa74d11d12cea19dcee3ceebe2" Jul 14 22:40:46.048508 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:40:46.048384 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6205e423-dgjrr_odh-ai-gateway-infra(241f1aff-eddb-44f2-9c9f-6e425b687e47)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" Jul 14 22:40:47.056644 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:47.056609 2578 scope.go:117] "RemoveContainer" containerID="7e158360c9e2067dcabc562c9b2b1cde35bfa2aa74d11d12cea19dcee3ceebe2" Jul 14 22:40:47.057002 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:40:47.056849 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6205e423-dgjrr_odh-ai-gateway-infra(241f1aff-eddb-44f2-9c9f-6e425b687e47)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" Jul 14 22:40:47.722283 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:47.722234 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:40:55.320432 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:55.319379 2578 patch_prober.go:28] interesting pod/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:33740->10.132.0.55:15021: read: connection reset by peer" start-of-body= Jul 14 22:40:55.320432 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:55.319457 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:33740->10.132.0.55:15021: read: connection reset by peer" Jul 14 22:40:56.106432 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:56.106395 2578 generic.go:356] "Generic (PLEG): container finished" podID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerID="a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7" exitCode=0 Jul 14 22:40:56.106608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:56.106467 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" event={"ID":"cc02ff7a-9669-4754-a03a-b28dbde7732c","Type":"ContainerDied","Data":"a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7"} Jul 14 22:40:56.106858 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:56.106841 2578 scope.go:117] "RemoveContainer" containerID="a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7" Jul 14 22:40:56.107822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:56.107792 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:40:56.107913 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:56.107861 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:40:56.107913 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:56.107892 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:40:57.113608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:57.113569 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" event={"ID":"cc02ff7a-9669-4754-a03a-b28dbde7732c","Type":"ContainerStarted","Data":"06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a"} Jul 14 22:40:57.288080 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:57.288041 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:40:57.288336 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:57.288315 2578 patch_prober.go:28] interesting pod/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 14 22:40:57.288400 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:57.288360 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 14 22:40:58.288675 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:58.288639 2578 patch_prober.go:28] interesting pod/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 14 22:40:58.289042 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:58.288704 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 14 22:40:59.288342 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:59.288306 2578 patch_prober.go:28] interesting pod/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 14 22:40:59.288519 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:59.288360 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 14 22:40:59.831728 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:40:59.831697 2578 scope.go:117] "RemoveContainer" containerID="7e158360c9e2067dcabc562c9b2b1cde35bfa2aa74d11d12cea19dcee3ceebe2" Jul 14 22:41:00.288354 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:00.288318 2578 patch_prober.go:28] interesting pod/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 14 22:41:00.288470 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:00.288370 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 14 22:41:01.135947 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:01.135855 2578 generic.go:356] "Generic (PLEG): container finished" podID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerID="dfcfeb664a43926a50c77ef5f12188fc0be583fa01d1e7f690ffe39d4e5d966f" exitCode=0 Jul 14 22:41:01.136407 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:01.135938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" event={"ID":"241f1aff-eddb-44f2-9c9f-6e425b687e47","Type":"ContainerDied","Data":"dfcfeb664a43926a50c77ef5f12188fc0be583fa01d1e7f690ffe39d4e5d966f"} Jul 14 22:41:01.136407 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:01.136023 2578 scope.go:117] "RemoveContainer" containerID="7e158360c9e2067dcabc562c9b2b1cde35bfa2aa74d11d12cea19dcee3ceebe2" Jul 14 22:41:01.292372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:01.292344 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:41:01.292803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:01.292789 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:41:01.293383 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:01.293366 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:41:02.173495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:02.173461 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr"] Jul 14 22:41:02.286636 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:02.286610 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" Jul 14 22:41:02.436267 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:02.436158 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241f1aff-eddb-44f2-9c9f-6e425b687e47-maas-api-service-ca\") pod \"241f1aff-eddb-44f2-9c9f-6e425b687e47\" (UID: \"241f1aff-eddb-44f2-9c9f-6e425b687e47\") " Jul 14 22:41:02.436582 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:02.436561 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241f1aff-eddb-44f2-9c9f-6e425b687e47-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "241f1aff-eddb-44f2-9c9f-6e425b687e47" (UID: "241f1aff-eddb-44f2-9c9f-6e425b687e47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:41:02.537168 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:02.537134 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241f1aff-eddb-44f2-9c9f-6e425b687e47-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:41:03.149938 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:03.149848 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" Jul 14 22:41:03.149938 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:03.149894 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr" event={"ID":"241f1aff-eddb-44f2-9c9f-6e425b687e47","Type":"ContainerDied","Data":"acffedaadc299b45410f04950c896e7608b419df0977b4120e580a4731c1a324"} Jul 14 22:41:03.150128 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:03.149945 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acffedaadc299b45410f04950c896e7608b419df0977b4120e580a4731c1a324" Jul 14 22:41:03.187800 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:03.187767 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr"] Jul 14 22:41:03.193935 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:03.193912 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-dgjrr"] Jul 14 22:41:04.837341 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:04.837307 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" path="/var/lib/kubelet/pods/241f1aff-eddb-44f2-9c9f-6e425b687e47/volumes" Jul 14 22:41:33.618535 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.618497 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km"] Jul 14 22:41:33.619150 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.619042 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerName="revoke-keys" Jul 14 22:41:33.619150 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.619061 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerName="revoke-keys" Jul 14 22:41:33.619150 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.619088 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerName="revoke-keys" Jul 14 22:41:33.619150 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.619097 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerName="revoke-keys" Jul 14 22:41:33.619150 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.619116 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" Jul 14 22:41:33.619150 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.619125 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ea73b0-3dbb-4ce3-af6a-ea58d550e361" containerName="istio-proxy" Jul 14 22:41:33.619519 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.619218 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerName="revoke-keys" Jul 14 22:41:33.619519 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.619232 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerName="revoke-keys" Jul 14 22:41:33.622459 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.622438 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" Jul 14 22:41:33.625813 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.625793 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:41:33.625918 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.625897 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:41:33.632278 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.632250 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km"] Jul 14 22:41:33.709541 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.709511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f4ff5-6535-4792-ba35-f30199cef036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6205e423-b66km\" (UID: \"7c5f4ff5-6535-4792-ba35-f30199cef036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" Jul 14 22:41:33.810224 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.810197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f4ff5-6535-4792-ba35-f30199cef036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6205e423-b66km\" (UID: \"7c5f4ff5-6535-4792-ba35-f30199cef036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" Jul 14 22:41:33.810789 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.810771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f4ff5-6535-4792-ba35-f30199cef036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6205e423-b66km\" (UID: \"7c5f4ff5-6535-4792-ba35-f30199cef036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" Jul 14 22:41:33.934985 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:33.934910 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" Jul 14 22:41:34.065708 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:34.065681 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km"] Jul 14 22:41:34.067157 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:41:34.067124 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5f4ff5_6535_4792_ba35_f30199cef036.slice/crio-53e692fc68a6f938ec33f92c591b4032f60aa8eaad2d8d0fbf3548e5991a0ba4 WatchSource:0}: Error finding container 53e692fc68a6f938ec33f92c591b4032f60aa8eaad2d8d0fbf3548e5991a0ba4: Status 404 returned error can't find the container with id 53e692fc68a6f938ec33f92c591b4032f60aa8eaad2d8d0fbf3548e5991a0ba4 Jul 14 22:41:34.306345 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:34.306317 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" event={"ID":"7c5f4ff5-6535-4792-ba35-f30199cef036","Type":"ContainerStarted","Data":"0bce73ef787e09a15d6127e03ba1fdef5bb0d0d4685ba19362190076b1664892"} Jul 14 22:41:34.306530 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:34.306366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" event={"ID":"7c5f4ff5-6535-4792-ba35-f30199cef036","Type":"ContainerStarted","Data":"53e692fc68a6f938ec33f92c591b4032f60aa8eaad2d8d0fbf3548e5991a0ba4"} Jul 14 22:41:34.324672 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:34.324627 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" podStartSLOduration=1.3246147719999999 podStartE2EDuration="1.324614772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:41:34.322777068 +0000 UTC m=+1960.135408709" watchObservedRunningTime="2026-07-14 22:41:34.324614772 +0000 UTC m=+1960.137246417" Jul 14 22:41:35.312622 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:35.312594 2578 generic.go:356] "Generic (PLEG): container finished" podID="7c5f4ff5-6535-4792-ba35-f30199cef036" containerID="0bce73ef787e09a15d6127e03ba1fdef5bb0d0d4685ba19362190076b1664892" exitCode=0 Jul 14 22:41:35.312977 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:35.312642 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" event={"ID":"7c5f4ff5-6535-4792-ba35-f30199cef036","Type":"ContainerDied","Data":"0bce73ef787e09a15d6127e03ba1fdef5bb0d0d4685ba19362190076b1664892"} Jul 14 22:41:36.448589 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:36.448564 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" Jul 14 22:41:36.533495 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:36.533468 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f4ff5-6535-4792-ba35-f30199cef036-maas-api-service-ca\") pod \"7c5f4ff5-6535-4792-ba35-f30199cef036\" (UID: \"7c5f4ff5-6535-4792-ba35-f30199cef036\") " Jul 14 22:41:36.533907 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:36.533875 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5f4ff5-6535-4792-ba35-f30199cef036-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c5f4ff5-6535-4792-ba35-f30199cef036" (UID: "7c5f4ff5-6535-4792-ba35-f30199cef036"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:41:36.635017 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:36.634926 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f4ff5-6535-4792-ba35-f30199cef036-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:41:37.323759 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:37.323724 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" Jul 14 22:41:37.323928 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:37.323757 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6205e423-b66km" event={"ID":"7c5f4ff5-6535-4792-ba35-f30199cef036","Type":"ContainerDied","Data":"53e692fc68a6f938ec33f92c591b4032f60aa8eaad2d8d0fbf3548e5991a0ba4"} Jul 14 22:41:37.323928 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:37.323806 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53e692fc68a6f938ec33f92c591b4032f60aa8eaad2d8d0fbf3548e5991a0ba4" Jul 14 22:41:46.890547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:46.890511 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:41:58.896907 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:58.896872 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2"] Jul 14 22:41:58.897333 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:41:58.897096 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" containerID="cri-o://06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a" gracePeriod=30 Jul 14 22:42:01.294415 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:01.294371 2578 patch_prober.go:28] interesting pod/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:42:01.294853 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:01.294441 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:42:04.146186 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.146158 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:42:04.280737 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.280704 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-token\") pod \"cc02ff7a-9669-4754-a03a-b28dbde7732c\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " Jul 14 22:42:04.280914 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.280757 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ff7a-9669-4754-a03a-b28dbde7732c-istiod-ca-cert\") pod \"cc02ff7a-9669-4754-a03a-b28dbde7732c\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " Jul 14 22:42:04.280914 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.280783 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-podinfo\") pod \"cc02ff7a-9669-4754-a03a-b28dbde7732c\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " Jul 14 22:42:04.280914 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.280824 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-socket\") pod \"cc02ff7a-9669-4754-a03a-b28dbde7732c\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " Jul 14 22:42:04.280914 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.280886 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc748\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-kube-api-access-hc748\") pod \"cc02ff7a-9669-4754-a03a-b28dbde7732c\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " Jul 14 22:42:04.281132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.280936 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-envoy\") pod \"cc02ff7a-9669-4754-a03a-b28dbde7732c\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " Jul 14 22:42:04.281132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.280973 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-certs\") pod \"cc02ff7a-9669-4754-a03a-b28dbde7732c\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " Jul 14 22:42:04.281132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.281004 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-credential-socket\") pod \"cc02ff7a-9669-4754-a03a-b28dbde7732c\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " Jul 14 22:42:04.281132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.281032 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-data\") pod \"cc02ff7a-9669-4754-a03a-b28dbde7732c\" (UID: \"cc02ff7a-9669-4754-a03a-b28dbde7732c\") " Jul 14 22:42:04.281364 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.281119 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc02ff7a-9669-4754-a03a-b28dbde7732c" (UID: "cc02ff7a-9669-4754-a03a-b28dbde7732c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:42:04.281430 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.281407 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc02ff7a-9669-4754-a03a-b28dbde7732c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc02ff7a-9669-4754-a03a-b28dbde7732c" (UID: "cc02ff7a-9669-4754-a03a-b28dbde7732c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:42:04.281430 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.281421 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:42:04.281693 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.281603 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc02ff7a-9669-4754-a03a-b28dbde7732c" (UID: "cc02ff7a-9669-4754-a03a-b28dbde7732c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:42:04.281826 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.281728 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc02ff7a-9669-4754-a03a-b28dbde7732c" (UID: "cc02ff7a-9669-4754-a03a-b28dbde7732c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:42:04.282051 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.282024 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc02ff7a-9669-4754-a03a-b28dbde7732c" (UID: "cc02ff7a-9669-4754-a03a-b28dbde7732c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:42:04.283102 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.283071 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc02ff7a-9669-4754-a03a-b28dbde7732c" (UID: "cc02ff7a-9669-4754-a03a-b28dbde7732c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:42:04.283185 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.283105 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-kube-api-access-hc748" (OuterVolumeSpecName: "kube-api-access-hc748") pod "cc02ff7a-9669-4754-a03a-b28dbde7732c" (UID: "cc02ff7a-9669-4754-a03a-b28dbde7732c"). InnerVolumeSpecName "kube-api-access-hc748". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:42:04.283370 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.283352 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc02ff7a-9669-4754-a03a-b28dbde7732c" (UID: "cc02ff7a-9669-4754-a03a-b28dbde7732c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:42:04.283370 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.283356 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc02ff7a-9669-4754-a03a-b28dbde7732c" (UID: "cc02ff7a-9669-4754-a03a-b28dbde7732c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:42:04.382905 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.382858 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:42:04.382905 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.382887 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ff7a-9669-4754-a03a-b28dbde7732c-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:42:04.382905 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.382898 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:42:04.382905 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.382907 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc748\" (UniqueName: \"kubernetes.io/projected/cc02ff7a-9669-4754-a03a-b28dbde7732c-kube-api-access-hc748\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:42:04.382905 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.382917 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:42:04.383340 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.382927 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:42:04.383340 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.382935 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:42:04.383340 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.382943 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ff7a-9669-4754-a03a-b28dbde7732c-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:42:04.467884 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.467797 2578 generic.go:356] "Generic (PLEG): container finished" podID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerID="06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a" exitCode=0 Jul 14 22:42:04.467884 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.467843 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" Jul 14 22:42:04.467884 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.467871 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" event={"ID":"cc02ff7a-9669-4754-a03a-b28dbde7732c","Type":"ContainerDied","Data":"06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a"} Jul 14 22:42:04.468212 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.467921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2" event={"ID":"cc02ff7a-9669-4754-a03a-b28dbde7732c","Type":"ContainerDied","Data":"869cce3a8b0bc564b6bd03673a7fa4d7295c2db313c5bce553382c519971b71a"} Jul 14 22:42:04.468212 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.467953 2578 scope.go:117] "RemoveContainer" containerID="06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a" Jul 14 22:42:04.478335 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.478311 2578 scope.go:117] "RemoveContainer" containerID="a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7" Jul 14 22:42:04.487049 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.487027 2578 scope.go:117] "RemoveContainer" containerID="06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a" Jul 14 22:42:04.487310 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:42:04.487288 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a\": container with ID starting with 06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a not found: ID does not exist" containerID="06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a" Jul 14 22:42:04.487363 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.487324 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a"} err="failed to get container status \"06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a\": rpc error: code = NotFound desc = could not find container \"06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a\": container with ID starting with 06bc1e973ba734522df57ba9727c367fe5b3c8147b918443af63ff0409d9ca8a not found: ID does not exist" Jul 14 22:42:04.487363 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.487348 2578 scope.go:117] "RemoveContainer" containerID="a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7" Jul 14 22:42:04.487602 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:42:04.487584 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7\": container with ID starting with a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7 not found: ID does not exist" containerID="a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7" Jul 14 22:42:04.487660 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.487607 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7"} err="failed to get container status \"a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7\": rpc error: code = NotFound desc = could not find container \"a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7\": container with ID starting with a9254ed986f4ce9ffa8b399a2bd25f99164a6f0d8d1817fc2f936e8c44b580b7 not found: ID does not exist" Jul 14 22:42:04.492988 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.492965 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2"] Jul 14 22:42:04.498747 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.498718 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6205e423-openshift-default-7d45db966c-srgh2"] Jul 14 22:42:04.839368 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:42:04.839333 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" path="/var/lib/kubelet/pods/cc02ff7a-9669-4754-a03a-b28dbde7732c/volumes" Jul 14 22:43:24.211687 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:24.211606 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:43:31.862366 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:31.862331 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:43:39.332911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.332878 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf"] Jul 14 22:43:39.333435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333280 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" Jul 14 22:43:39.333435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333293 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" Jul 14 22:43:39.333435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333304 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" Jul 14 22:43:39.333435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333309 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" Jul 14 22:43:39.333435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333324 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5f4ff5-6535-4792-ba35-f30199cef036" containerName="revoke-keys" Jul 14 22:43:39.333435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333331 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5f4ff5-6535-4792-ba35-f30199cef036" containerName="revoke-keys" Jul 14 22:43:39.333435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333338 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerName="revoke-keys" Jul 14 22:43:39.333435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333343 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerName="revoke-keys" Jul 14 22:43:39.333862 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333444 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c5f4ff5-6535-4792-ba35-f30199cef036" containerName="revoke-keys" Jul 14 22:43:39.333862 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333455 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" Jul 14 22:43:39.333862 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333462 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="241f1aff-eddb-44f2-9c9f-6e425b687e47" containerName="revoke-keys" Jul 14 22:43:39.333862 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.333469 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc02ff7a-9669-4754-a03a-b28dbde7732c" containerName="istio-proxy" Jul 14 22:43:39.336578 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.336557 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.339881 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.339861 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-de2bcd69-openshift-default-dockercfg-spxd8\"" Jul 14 22:43:39.365490 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.365464 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf"] Jul 14 22:43:39.409006 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.408979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4c479-f505-42bb-9de9-dceea1748906-istio-podinfo\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.409133 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.409033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-data\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.409133 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.409112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-istio-token\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.409210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.409151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zqsc\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-kube-api-access-4zqsc\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.409278 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.409226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-credential-socket\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.409325 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.409285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-socket\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.409325 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.409318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-certs\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.409400 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.409341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-envoy\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.409400 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.409366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4c479-f505-42bb-9de9-dceea1748906-istiod-ca-cert\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.509827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.509791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-data\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.509827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.509828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-istio-token\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.509853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqsc\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-kube-api-access-4zqsc\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.509887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-credential-socket\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.509909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-socket\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.509939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-certs\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.509960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-envoy\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.509983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4c479-f505-42bb-9de9-dceea1748906-istiod-ca-cert\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510434 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.510097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4c479-f505-42bb-9de9-dceea1748906-istio-podinfo\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510434 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.510229 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-data\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510434 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.510306 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-credential-socket\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510623 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.510599 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-certs\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510677 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.510635 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-socket\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.510787 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.510766 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4c479-f505-42bb-9de9-dceea1748906-istiod-ca-cert\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.512570 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.512541 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-envoy\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.512570 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.512564 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4c479-f505-42bb-9de9-dceea1748906-istio-podinfo\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.519559 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.519537 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqsc\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-kube-api-access-4zqsc\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.519876 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.519860 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-istio-token\") pod \"e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.648288 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.648181 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:39.859610 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.859585 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf"] Jul 14 22:43:39.861232 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:43:39.861197 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d4c479_f505_42bb_9de9_dceea1748906.slice/crio-e70dd38a5523250638d6a272af1c59a43d3b0e69f82bf57c099289ab93951727 WatchSource:0}: Error finding container e70dd38a5523250638d6a272af1c59a43d3b0e69f82bf57c099289ab93951727: Status 404 returned error can't find the container with id e70dd38a5523250638d6a272af1c59a43d3b0e69f82bf57c099289ab93951727 Jul 14 22:43:39.863472 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.863432 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:43:39.863578 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.863515 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:43:39.863578 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.863558 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:43:39.961927 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.961886 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" event={"ID":"28d4c479-f505-42bb-9de9-dceea1748906","Type":"ContainerStarted","Data":"21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48"} Jul 14 22:43:39.962073 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:39.961953 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" event={"ID":"28d4c479-f505-42bb-9de9-dceea1748906","Type":"ContainerStarted","Data":"e70dd38a5523250638d6a272af1c59a43d3b0e69f82bf57c099289ab93951727"} Jul 14 22:43:40.003606 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:40.003546 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" podStartSLOduration=1.003525981 podStartE2EDuration="1.003525981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:43:40.000439706 +0000 UTC m=+2085.813071367" watchObservedRunningTime="2026-07-14 22:43:40.003525981 +0000 UTC m=+2085.816157621" Jul 14 22:43:40.054292 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:40.054257 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:43:40.649138 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:40.649103 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:40.654099 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:40.654076 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:40.967344 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:40.967266 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:40.968267 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:40.968229 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:47.999236 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:47.999202 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:43:54.661634 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.661603 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp"] Jul 14 22:43:54.665679 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.665660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" Jul 14 22:43:54.668990 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.668966 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:43:54.669260 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.669220 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:43:54.678224 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.678205 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp"] Jul 14 22:43:54.751208 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.751180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a017-24c6-403f-9c72-fb35a0815c31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp\" (UID: \"96d3a017-24c6-403f-9c72-fb35a0815c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" Jul 14 22:43:54.808733 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.808709 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:43:54.811636 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.811613 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:43:54.852273 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.852217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a017-24c6-403f-9c72-fb35a0815c31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp\" (UID: \"96d3a017-24c6-403f-9c72-fb35a0815c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" Jul 14 22:43:54.852827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.852807 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a017-24c6-403f-9c72-fb35a0815c31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp\" (UID: \"96d3a017-24c6-403f-9c72-fb35a0815c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" Jul 14 22:43:54.976361 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:54.976277 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" Jul 14 22:43:55.126361 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:55.126334 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp"] Jul 14 22:43:55.128577 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:43:55.128548 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d3a017_24c6_403f_9c72_fb35a0815c31.slice/crio-b13ea157d12b1bddf6933a57a0ddefbaf46e81a81c56922fac6449b6cdc1290a WatchSource:0}: Error finding container b13ea157d12b1bddf6933a57a0ddefbaf46e81a81c56922fac6449b6cdc1290a: Status 404 returned error can't find the container with id b13ea157d12b1bddf6933a57a0ddefbaf46e81a81c56922fac6449b6cdc1290a Jul 14 22:43:55.969025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:55.968996 2578 patch_prober.go:28] interesting pod/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 22:43:55.969385 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:55.969044 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 22:43:56.049855 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:56.049826 2578 generic.go:356] "Generic (PLEG): container finished" podID="28d4c479-f505-42bb-9de9-dceea1748906" containerID="21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48" exitCode=0 Jul 14 22:43:56.050000 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:56.049901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" event={"ID":"28d4c479-f505-42bb-9de9-dceea1748906","Type":"ContainerDied","Data":"21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48"} Jul 14 22:43:56.050328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:56.050296 2578 scope.go:117] "RemoveContainer" containerID="21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48" Jul 14 22:43:56.051460 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:56.051423 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:43:56.051575 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:56.051508 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:43:56.051575 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:56.051550 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:43:56.051682 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:56.051611 2578 generic.go:356] "Generic (PLEG): container finished" podID="96d3a017-24c6-403f-9c72-fb35a0815c31" containerID="ada55ee64d9a7262e70a5e728bd5b8245a19e9707063b5aac80f0c2fddcf1ca6" exitCode=0 Jul 14 22:43:56.051739 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:56.051690 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" event={"ID":"96d3a017-24c6-403f-9c72-fb35a0815c31","Type":"ContainerDied","Data":"ada55ee64d9a7262e70a5e728bd5b8245a19e9707063b5aac80f0c2fddcf1ca6"} Jul 14 22:43:56.051791 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:56.051746 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" event={"ID":"96d3a017-24c6-403f-9c72-fb35a0815c31","Type":"ContainerStarted","Data":"b13ea157d12b1bddf6933a57a0ddefbaf46e81a81c56922fac6449b6cdc1290a"} Jul 14 22:43:57.059832 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:57.059796 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" event={"ID":"28d4c479-f505-42bb-9de9-dceea1748906","Type":"ContainerStarted","Data":"4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40"} Jul 14 22:43:57.219752 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:57.219731 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" Jul 14 22:43:57.375315 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:57.375191 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a017-24c6-403f-9c72-fb35a0815c31-maas-api-service-ca\") pod \"96d3a017-24c6-403f-9c72-fb35a0815c31\" (UID: \"96d3a017-24c6-403f-9c72-fb35a0815c31\") " Jul 14 22:43:57.375616 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:57.375593 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d3a017-24c6-403f-9c72-fb35a0815c31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96d3a017-24c6-403f-9c72-fb35a0815c31" (UID: "96d3a017-24c6-403f-9c72-fb35a0815c31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:43:57.483194 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:57.483151 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d3a017-24c6-403f-9c72-fb35a0815c31-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:43:57.649511 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:57.649412 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:43:57.649692 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:57.649580 2578 patch_prober.go:28] interesting pod/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 22:43:57.649692 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:57.649629 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 22:43:58.066728 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:58.066698 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" Jul 14 22:43:58.067112 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:58.066733 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp" event={"ID":"96d3a017-24c6-403f-9c72-fb35a0815c31","Type":"ContainerDied","Data":"b13ea157d12b1bddf6933a57a0ddefbaf46e81a81c56922fac6449b6cdc1290a"} Jul 14 22:43:58.067112 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:58.066780 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b13ea157d12b1bddf6933a57a0ddefbaf46e81a81c56922fac6449b6cdc1290a" Jul 14 22:43:58.649096 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:58.649049 2578 patch_prober.go:28] interesting pod/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 22:43:58.649322 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:58.649118 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 22:43:59.649660 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:59.649598 2578 patch_prober.go:28] interesting pod/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 22:43:59.650173 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:43:59.649692 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 22:44:00.649138 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:00.649099 2578 patch_prober.go:28] interesting pod/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 22:44:00.649438 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:00.649408 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 22:44:01.653020 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:01.652986 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:44:01.653506 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:01.653268 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:44:01.653920 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:01.653906 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:44:08.102688 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:08.102649 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:44:20.603195 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:20.603160 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf"] Jul 14 22:44:20.603589 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:20.603376 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" containerID="cri-o://4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40" gracePeriod=30 Jul 14 22:44:25.852760 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.852734 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:44:25.935446 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935372 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4c479-f505-42bb-9de9-dceea1748906-istio-podinfo\") pod \"28d4c479-f505-42bb-9de9-dceea1748906\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " Jul 14 22:44:25.935446 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935404 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-certs\") pod \"28d4c479-f505-42bb-9de9-dceea1748906\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " Jul 14 22:44:25.935446 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935444 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-socket\") pod \"28d4c479-f505-42bb-9de9-dceea1748906\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " Jul 14 22:44:25.935659 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935553 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-envoy\") pod \"28d4c479-f505-42bb-9de9-dceea1748906\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " Jul 14 22:44:25.935659 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935619 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-data\") pod \"28d4c479-f505-42bb-9de9-dceea1748906\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " Jul 14 22:44:25.935764 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935666 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "28d4c479-f505-42bb-9de9-dceea1748906" (UID: "28d4c479-f505-42bb-9de9-dceea1748906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:44:25.935764 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935674 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-credential-socket\") pod \"28d4c479-f505-42bb-9de9-dceea1748906\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " Jul 14 22:44:25.935764 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935755 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "28d4c479-f505-42bb-9de9-dceea1748906" (UID: "28d4c479-f505-42bb-9de9-dceea1748906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:44:25.935929 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935782 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4c479-f505-42bb-9de9-dceea1748906-istiod-ca-cert\") pod \"28d4c479-f505-42bb-9de9-dceea1748906\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " Jul 14 22:44:25.935929 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935817 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-istio-token\") pod \"28d4c479-f505-42bb-9de9-dceea1748906\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " Jul 14 22:44:25.935929 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935843 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zqsc\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-kube-api-access-4zqsc\") pod \"28d4c479-f505-42bb-9de9-dceea1748906\" (UID: \"28d4c479-f505-42bb-9de9-dceea1748906\") " Jul 14 22:44:25.935929 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.935881 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "28d4c479-f505-42bb-9de9-dceea1748906" (UID: "28d4c479-f505-42bb-9de9-dceea1748906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:44:25.936234 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.936161 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d4c479-f505-42bb-9de9-dceea1748906-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "28d4c479-f505-42bb-9de9-dceea1748906" (UID: "28d4c479-f505-42bb-9de9-dceea1748906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:44:25.936234 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.936171 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:44:25.936234 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.936214 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:44:25.936424 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.936232 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:44:25.936599 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.936574 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-data" (OuterVolumeSpecName: "istio-data") pod "28d4c479-f505-42bb-9de9-dceea1748906" (UID: "28d4c479-f505-42bb-9de9-dceea1748906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:44:25.937692 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.937670 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "28d4c479-f505-42bb-9de9-dceea1748906" (UID: "28d4c479-f505-42bb-9de9-dceea1748906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:44:25.937692 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.937678 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28d4c479-f505-42bb-9de9-dceea1748906-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "28d4c479-f505-42bb-9de9-dceea1748906" (UID: "28d4c479-f505-42bb-9de9-dceea1748906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:44:25.938192 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.938174 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-istio-token" (OuterVolumeSpecName: "istio-token") pod "28d4c479-f505-42bb-9de9-dceea1748906" (UID: "28d4c479-f505-42bb-9de9-dceea1748906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:44:25.938279 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:25.938190 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-kube-api-access-4zqsc" (OuterVolumeSpecName: "kube-api-access-4zqsc") pod "28d4c479-f505-42bb-9de9-dceea1748906" (UID: "28d4c479-f505-42bb-9de9-dceea1748906"). InnerVolumeSpecName "kube-api-access-4zqsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:44:26.037654 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.037611 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4c479-f505-42bb-9de9-dceea1748906-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:44:26.037654 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.037646 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:44:26.037654 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.037658 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zqsc\" (UniqueName: \"kubernetes.io/projected/28d4c479-f505-42bb-9de9-dceea1748906-kube-api-access-4zqsc\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:44:26.037855 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.037671 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4c479-f505-42bb-9de9-dceea1748906-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:44:26.037855 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.037682 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:44:26.037855 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.037694 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4c479-f505-42bb-9de9-dceea1748906-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:44:26.220479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.220395 2578 generic.go:356] "Generic (PLEG): container finished" podID="28d4c479-f505-42bb-9de9-dceea1748906" containerID="4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40" exitCode=0 Jul 14 22:44:26.220479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.220442 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" Jul 14 22:44:26.220646 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.220476 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" event={"ID":"28d4c479-f505-42bb-9de9-dceea1748906","Type":"ContainerDied","Data":"4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40"} Jul 14 22:44:26.220646 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.220529 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf" event={"ID":"28d4c479-f505-42bb-9de9-dceea1748906","Type":"ContainerDied","Data":"e70dd38a5523250638d6a272af1c59a43d3b0e69f82bf57c099289ab93951727"} Jul 14 22:44:26.220646 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.220553 2578 scope.go:117] "RemoveContainer" containerID="4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40" Jul 14 22:44:26.230044 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.230026 2578 scope.go:117] "RemoveContainer" containerID="21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48" Jul 14 22:44:26.237991 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.237976 2578 scope.go:117] "RemoveContainer" containerID="4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40" Jul 14 22:44:26.238233 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:44:26.238214 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40\": container with ID starting with 4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40 not found: ID does not exist" containerID="4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40" Jul 14 22:44:26.238338 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.238262 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40"} err="failed to get container status \"4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40\": rpc error: code = NotFound desc = could not find container \"4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40\": container with ID starting with 4b117722cd1de43f08a632ce06346c01216bde30d06c8250268ab1ba95d9ce40 not found: ID does not exist" Jul 14 22:44:26.238338 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.238285 2578 scope.go:117] "RemoveContainer" containerID="21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48" Jul 14 22:44:26.238533 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:44:26.238509 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48\": container with ID starting with 21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48 not found: ID does not exist" containerID="21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48" Jul 14 22:44:26.238579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.238542 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48"} err="failed to get container status \"21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48\": rpc error: code = NotFound desc = could not find container \"21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48\": container with ID starting with 21f74ee5b4a8efa88d8dba85b9769f419c8b0c32137c8bc2b73f11f4593efd48 not found: ID does not exist" Jul 14 22:44:26.243276 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.243256 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf"] Jul 14 22:44:26.247370 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.247351 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-de2bcd69-openshift-default-bd6bd6f59-2gvqf"] Jul 14 22:44:26.844300 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:26.844265 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28d4c479-f505-42bb-9de9-dceea1748906" path="/var/lib/kubelet/pods/28d4c479-f505-42bb-9de9-dceea1748906/volumes" Jul 14 22:44:41.727728 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.727695 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff"] Jul 14 22:44:41.728354 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.728335 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" Jul 14 22:44:41.728447 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.728356 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" Jul 14 22:44:41.728447 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.728371 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d3a017-24c6-403f-9c72-fb35a0815c31" containerName="revoke-keys" Jul 14 22:44:41.728447 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.728379 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d3a017-24c6-403f-9c72-fb35a0815c31" containerName="revoke-keys" Jul 14 22:44:41.728447 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.728401 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" Jul 14 22:44:41.728447 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.728411 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" Jul 14 22:44:41.728692 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.728551 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d3a017-24c6-403f-9c72-fb35a0815c31" containerName="revoke-keys" Jul 14 22:44:41.728692 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.728567 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" Jul 14 22:44:41.731546 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.731518 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.735669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.735651 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7a211570-openshift-default-dockercfg-lw5zp\"" Jul 14 22:44:41.849459 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.849431 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff"] Jul 14 22:44:41.878913 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.878872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgvv\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-kube-api-access-ncgvv\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.879091 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.878949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istiod-ca-cert\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.879091 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.878985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-socket\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.879091 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.879014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-envoy\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.879091 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.879045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-token\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.879257 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.879119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-data\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.879257 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.879172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-certs\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.879257 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.879207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-podinfo\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.879378 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.879267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-credential-socket\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.980586 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-certs\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.980586 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-podinfo\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.980816 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-credential-socket\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.980816 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgvv\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-kube-api-access-ncgvv\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.980816 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istiod-ca-cert\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.980816 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-socket\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.980816 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-envoy\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.980816 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-token\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.981132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-data\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.981132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.980872 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-certs\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.981132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.981126 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-data\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.981653 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.981177 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-credential-socket\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.981653 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.981376 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-socket\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.981653 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.981504 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istiod-ca-cert\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.984397 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.984374 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-podinfo\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.984994 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.984970 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-envoy\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:41.998418 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:41.998392 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-token\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:42.004531 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:42.004501 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgvv\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-kube-api-access-ncgvv\") pod \"e2e-mt-7a211570-openshift-default-d8955b6d7-stwff\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:42.042461 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:42.042432 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:42.214511 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:42.214477 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff"] Jul 14 22:44:42.216580 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:44:42.216545 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae41d9a_e780_4ac5_aa78_0e74057526eb.slice/crio-34309ac96d5755d03e6e9815592f2e53281eba0c5728b9952053c31a844a971f WatchSource:0}: Error finding container 34309ac96d5755d03e6e9815592f2e53281eba0c5728b9952053c31a844a971f: Status 404 returned error can't find the container with id 34309ac96d5755d03e6e9815592f2e53281eba0c5728b9952053c31a844a971f Jul 14 22:44:42.218743 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:42.218705 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:44:42.218845 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:42.218792 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:44:42.218845 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:42.218837 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:44:42.308214 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:42.308173 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" event={"ID":"4ae41d9a-e780-4ac5-aa78-0e74057526eb","Type":"ContainerStarted","Data":"88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5"} Jul 14 22:44:42.308411 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:42.308254 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" event={"ID":"4ae41d9a-e780-4ac5-aa78-0e74057526eb","Type":"ContainerStarted","Data":"34309ac96d5755d03e6e9815592f2e53281eba0c5728b9952053c31a844a971f"} Jul 14 22:44:42.331888 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:42.331802 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" podStartSLOduration=1.331780741 podStartE2EDuration="1.331780741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:44:42.329654941 +0000 UTC m=+2148.142286584" watchObservedRunningTime="2026-07-14 22:44:42.331780741 +0000 UTC m=+2148.144412382" Jul 14 22:44:43.043370 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:43.043339 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:43.048400 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:43.048374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:43.313443 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:43.313414 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:43.314189 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:43.314170 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:47.193139 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:47.193065 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:44:54.900552 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:54.900524 2578 scope.go:117] "RemoveContainer" containerID="b4a9bb711a70db26d33e386c778fb33a12091a21532700aca66ec3af9079f451" Jul 14 22:44:55.394299 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:55.394267 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerID="88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5" exitCode=0 Jul 14 22:44:55.394541 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:55.394347 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" event={"ID":"4ae41d9a-e780-4ac5-aa78-0e74057526eb","Type":"ContainerDied","Data":"88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5"} Jul 14 22:44:55.394726 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:55.394712 2578 scope.go:117] "RemoveContainer" containerID="88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5" Jul 14 22:44:55.395667 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:55.395640 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:44:55.395737 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:55.395713 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:44:55.395773 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:55.395747 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:44:56.401985 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:56.401953 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" event={"ID":"4ae41d9a-e780-4ac5-aa78-0e74057526eb","Type":"ContainerStarted","Data":"247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f"} Jul 14 22:44:57.044734 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:57.044692 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:57.044942 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:57.044828 2578 patch_prober.go:28] interesting pod/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 14 22:44:57.044942 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:57.044872 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 14 22:44:58.043216 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:58.043171 2578 patch_prober.go:28] interesting pod/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 14 22:44:58.043756 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:58.043263 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 14 22:44:58.314227 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:58.314197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:44:59.043065 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:59.043019 2578 patch_prober.go:28] interesting pod/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 14 22:44:59.043294 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:44:59.043090 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 14 22:45:00.043479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.043435 2578 patch_prober.go:28] interesting pod/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 14 22:45:00.043958 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.043540 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 14 22:45:00.144493 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.144456 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls"] Jul 14 22:45:00.145183 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.145158 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="28d4c479-f505-42bb-9de9-dceea1748906" containerName="istio-proxy" Jul 14 22:45:00.148695 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.148670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls" Jul 14 22:45:00.149817 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.149727 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g"] Jul 14 22:45:00.151969 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.151945 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:45:00.161374 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.160602 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g"] Jul 14 22:45:00.161374 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.160703 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g" Jul 14 22:45:00.161497 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.161379 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls"] Jul 14 22:45:00.165690 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.165670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls" Jul 14 22:45:00.175939 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.175920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g" Jul 14 22:45:00.352580 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.348176 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls"] Jul 14 22:45:00.404355 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.404330 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g"] Jul 14 22:45:00.406264 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:45:00.406226 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90643614_5547_4ec1_99d9_9799c66a7679.slice/crio-602481cabd942bb8bab1ff502e6a3fa795bc26fa4c21696326bf6d750af83285 WatchSource:0}: Error finding container 602481cabd942bb8bab1ff502e6a3fa795bc26fa4c21696326bf6d750af83285: Status 404 returned error can't find the container with id 602481cabd942bb8bab1ff502e6a3fa795bc26fa4c21696326bf6d750af83285 Jul 14 22:45:00.424583 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.424557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls" event={"ID":"7d960c41-0e6b-4d06-80aa-5195222f3fdd","Type":"ContainerStarted","Data":"441e636fa56b6a5a27ea7d9f5629cb9183ec334cc59dda1cb4ce3f2ddc89bee9"} Jul 14 22:45:00.425832 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:00.425810 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g" event={"ID":"90643614-5547-4ec1-99d9-9799c66a7679","Type":"ContainerStarted","Data":"602481cabd942bb8bab1ff502e6a3fa795bc26fa4c21696326bf6d750af83285"} Jul 14 22:45:01.047302 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:01.047268 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:45:01.048307 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:01.048286 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:45:01.433183 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:01.433147 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls" event={"ID":"7d960c41-0e6b-4d06-80aa-5195222f3fdd","Type":"ContainerStarted","Data":"b7ad04a1985f49b04facbc19c7bc8c25eaa3f3560399f433bf4c72ae1e06b9c7"} Jul 14 22:45:01.434969 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:01.434933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g" event={"ID":"90643614-5547-4ec1-99d9-9799c66a7679","Type":"ContainerStarted","Data":"6e71bb9bd2e51b6f7030dca7b28a3418f80b4ff09019b6b6dbf563b786e2d86b"} Jul 14 22:45:01.449944 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:01.449748 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls" podStartSLOduration=1.449731964 podStartE2EDuration="1.449731964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:45:01.449090102 +0000 UTC m=+2167.261721747" watchObservedRunningTime="2026-07-14 22:45:01.449731964 +0000 UTC m=+2167.262363603" Jul 14 22:45:01.464315 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:01.464267 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g" podStartSLOduration=1.464224517 podStartE2EDuration="1.464224517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:45:01.463724399 +0000 UTC m=+2167.276356040" watchObservedRunningTime="2026-07-14 22:45:01.464224517 +0000 UTC m=+2167.276856178" Jul 14 22:45:02.442073 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:02.442037 2578 generic.go:356] "Generic (PLEG): container finished" podID="7d960c41-0e6b-4d06-80aa-5195222f3fdd" containerID="b7ad04a1985f49b04facbc19c7bc8c25eaa3f3560399f433bf4c72ae1e06b9c7" exitCode=0 Jul 14 22:45:02.442510 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:02.442124 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls" event={"ID":"7d960c41-0e6b-4d06-80aa-5195222f3fdd","Type":"ContainerDied","Data":"b7ad04a1985f49b04facbc19c7bc8c25eaa3f3560399f433bf4c72ae1e06b9c7"} Jul 14 22:45:02.443596 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:02.443573 2578 generic.go:356] "Generic (PLEG): container finished" podID="90643614-5547-4ec1-99d9-9799c66a7679" containerID="6e71bb9bd2e51b6f7030dca7b28a3418f80b4ff09019b6b6dbf563b786e2d86b" exitCode=0 Jul 14 22:45:02.443653 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:02.443613 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g" event={"ID":"90643614-5547-4ec1-99d9-9799c66a7679","Type":"ContainerDied","Data":"6e71bb9bd2e51b6f7030dca7b28a3418f80b4ff09019b6b6dbf563b786e2d86b"} Jul 14 22:45:03.599635 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:03.599604 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls" Jul 14 22:45:03.602524 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:03.602504 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g" Jul 14 22:45:04.455703 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:04.455677 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls" Jul 14 22:45:04.455885 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:04.455711 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls" event={"ID":"7d960c41-0e6b-4d06-80aa-5195222f3fdd","Type":"ContainerDied","Data":"441e636fa56b6a5a27ea7d9f5629cb9183ec334cc59dda1cb4ce3f2ddc89bee9"} Jul 14 22:45:04.455885 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:04.455758 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="441e636fa56b6a5a27ea7d9f5629cb9183ec334cc59dda1cb4ce3f2ddc89bee9" Jul 14 22:45:04.457112 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:04.457098 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g" Jul 14 22:45:04.457211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:04.457124 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734485-6lb7g" event={"ID":"90643614-5547-4ec1-99d9-9799c66a7679","Type":"ContainerDied","Data":"602481cabd942bb8bab1ff502e6a3fa795bc26fa4c21696326bf6d750af83285"} Jul 14 22:45:04.457211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:04.457168 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="602481cabd942bb8bab1ff502e6a3fa795bc26fa4c21696326bf6d750af83285" Jul 14 22:45:13.088890 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:13.088858 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:45:31.594152 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:31.594114 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:45:37.720067 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.720030 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n"] Jul 14 22:45:37.720470 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.720428 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90643614-5547-4ec1-99d9-9799c66a7679" containerName="cleanup" Jul 14 22:45:37.720470 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.720440 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90643614-5547-4ec1-99d9-9799c66a7679" containerName="cleanup" Jul 14 22:45:37.720470 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.720455 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d960c41-0e6b-4d06-80aa-5195222f3fdd" containerName="cleanup" Jul 14 22:45:37.720470 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.720461 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d960c41-0e6b-4d06-80aa-5195222f3fdd" containerName="cleanup" Jul 14 22:45:37.720611 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.720521 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d960c41-0e6b-4d06-80aa-5195222f3fdd" containerName="cleanup" Jul 14 22:45:37.720611 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.720531 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="90643614-5547-4ec1-99d9-9799c66a7679" containerName="cleanup" Jul 14 22:45:37.723758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.723739 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" Jul 14 22:45:37.726520 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.726497 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:45:37.726634 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.726538 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:45:37.730861 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.730838 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n"] Jul 14 22:45:37.914265 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:37.914200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c838f78f-ff88-4e13-b162-ed646c8665c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a211570-28v7n\" (UID: \"c838f78f-ff88-4e13-b162-ed646c8665c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" Jul 14 22:45:38.015727 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:38.015631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c838f78f-ff88-4e13-b162-ed646c8665c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a211570-28v7n\" (UID: \"c838f78f-ff88-4e13-b162-ed646c8665c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" Jul 14 22:45:38.016289 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:38.016233 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c838f78f-ff88-4e13-b162-ed646c8665c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a211570-28v7n\" (UID: \"c838f78f-ff88-4e13-b162-ed646c8665c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" Jul 14 22:45:38.036613 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:38.036590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" Jul 14 22:45:38.380491 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:38.380458 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n"] Jul 14 22:45:38.382032 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:45:38.382006 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc838f78f_ff88_4e13_b162_ed646c8665c5.slice/crio-054d95959aace480e8b10ab53ce3ca8b888a1c97a29a16ecf8c0415d9ec622c7 WatchSource:0}: Error finding container 054d95959aace480e8b10ab53ce3ca8b888a1c97a29a16ecf8c0415d9ec622c7: Status 404 returned error can't find the container with id 054d95959aace480e8b10ab53ce3ca8b888a1c97a29a16ecf8c0415d9ec622c7 Jul 14 22:45:38.636864 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:38.636775 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" event={"ID":"c838f78f-ff88-4e13-b162-ed646c8665c5","Type":"ContainerStarted","Data":"7131014c4e76b634b64c7e672bd84e731865cb8ef9e34ee2beb881b5cccf2a78"} Jul 14 22:45:38.636864 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:38.636832 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" event={"ID":"c838f78f-ff88-4e13-b162-ed646c8665c5","Type":"ContainerStarted","Data":"054d95959aace480e8b10ab53ce3ca8b888a1c97a29a16ecf8c0415d9ec622c7"} Jul 14 22:45:38.655173 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:38.655121 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" podStartSLOduration=1.6551045979999999 podStartE2EDuration="1.655104598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:45:38.654433307 +0000 UTC m=+2204.467064947" watchObservedRunningTime="2026-07-14 22:45:38.655104598 +0000 UTC m=+2204.467736240" Jul 14 22:45:39.643375 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:39.643339 2578 generic.go:356] "Generic (PLEG): container finished" podID="c838f78f-ff88-4e13-b162-ed646c8665c5" containerID="7131014c4e76b634b64c7e672bd84e731865cb8ef9e34ee2beb881b5cccf2a78" exitCode=0 Jul 14 22:45:39.643812 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:39.643424 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" event={"ID":"c838f78f-ff88-4e13-b162-ed646c8665c5","Type":"ContainerDied","Data":"7131014c4e76b634b64c7e672bd84e731865cb8ef9e34ee2beb881b5cccf2a78"} Jul 14 22:45:40.790644 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:40.790616 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" Jul 14 22:45:40.943471 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:40.943383 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c838f78f-ff88-4e13-b162-ed646c8665c5-maas-api-service-ca\") pod \"c838f78f-ff88-4e13-b162-ed646c8665c5\" (UID: \"c838f78f-ff88-4e13-b162-ed646c8665c5\") " Jul 14 22:45:40.943732 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:40.943704 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c838f78f-ff88-4e13-b162-ed646c8665c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c838f78f-ff88-4e13-b162-ed646c8665c5" (UID: "c838f78f-ff88-4e13-b162-ed646c8665c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:45:41.044260 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:41.044209 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c838f78f-ff88-4e13-b162-ed646c8665c5-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:45:41.656328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:41.656297 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" Jul 14 22:45:41.656328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:41.656301 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a211570-28v7n" event={"ID":"c838f78f-ff88-4e13-b162-ed646c8665c5","Type":"ContainerDied","Data":"054d95959aace480e8b10ab53ce3ca8b888a1c97a29a16ecf8c0415d9ec622c7"} Jul 14 22:45:41.656551 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:41.656384 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="054d95959aace480e8b10ab53ce3ca8b888a1c97a29a16ecf8c0415d9ec622c7" Jul 14 22:45:48.283622 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:48.283592 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls"] Jul 14 22:45:48.286599 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:48.286577 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7a211570-29734485-wxbls"] Jul 14 22:45:48.836717 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:48.836677 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d960c41-0e6b-4d06-80aa-5195222f3fdd" path="/var/lib/kubelet/pods/7d960c41-0e6b-4d06-80aa-5195222f3fdd/volumes" Jul 14 22:45:51.601421 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:51.601386 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:45:54.929422 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:45:54.929391 2578 scope.go:117] "RemoveContainer" containerID="0397dadcf16ad54fe7624a76dfea4a117a0d92c15416b2b3b0da07c8ea0274b8" Jul 14 22:46:45.170103 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:45.170023 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff"] Jul 14 22:46:45.170649 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:45.170261 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" containerID="cri-o://247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f" gracePeriod=30 Jul 14 22:46:46.050106 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:46.050062 2578 patch_prober.go:28] interesting pod/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:46:46.050300 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:46.050138 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:46:49.595422 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:49.595335 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:46:50.432620 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.432557 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:46:50.544635 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544603 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-podinfo\") pod \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " Jul 14 22:46:50.544793 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544659 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-socket\") pod \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " Jul 14 22:46:50.544793 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544687 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-envoy\") pod \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " Jul 14 22:46:50.544793 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544743 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncgvv\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-kube-api-access-ncgvv\") pod \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " Jul 14 22:46:50.544793 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544769 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-token\") pod \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " Jul 14 22:46:50.545025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544850 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istiod-ca-cert\") pod \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " Jul 14 22:46:50.545025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544892 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-data\") pod \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " Jul 14 22:46:50.545025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544926 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-certs\") pod \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " Jul 14 22:46:50.545025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544953 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-credential-socket\") pod \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\" (UID: \"4ae41d9a-e780-4ac5-aa78-0e74057526eb\") " Jul 14 22:46:50.545025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.544994 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ae41d9a-e780-4ac5-aa78-0e74057526eb" (UID: "4ae41d9a-e780-4ac5-aa78-0e74057526eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:46:50.545327 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.545172 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ae41d9a-e780-4ac5-aa78-0e74057526eb" (UID: "4ae41d9a-e780-4ac5-aa78-0e74057526eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:46:50.545327 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.545290 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:46:50.545327 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.545312 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:46:50.545490 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.545327 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ae41d9a-e780-4ac5-aa78-0e74057526eb" (UID: "4ae41d9a-e780-4ac5-aa78-0e74057526eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:46:50.545686 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.545635 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ae41d9a-e780-4ac5-aa78-0e74057526eb" (UID: "4ae41d9a-e780-4ac5-aa78-0e74057526eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:46:50.545825 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.545748 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ae41d9a-e780-4ac5-aa78-0e74057526eb" (UID: "4ae41d9a-e780-4ac5-aa78-0e74057526eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:46:50.547052 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.547029 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ae41d9a-e780-4ac5-aa78-0e74057526eb" (UID: "4ae41d9a-e780-4ac5-aa78-0e74057526eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:46:50.547052 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.547048 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-kube-api-access-ncgvv" (OuterVolumeSpecName: "kube-api-access-ncgvv") pod "4ae41d9a-e780-4ac5-aa78-0e74057526eb" (UID: "4ae41d9a-e780-4ac5-aa78-0e74057526eb"). InnerVolumeSpecName "kube-api-access-ncgvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:46:50.547357 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.547332 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ae41d9a-e780-4ac5-aa78-0e74057526eb" (UID: "4ae41d9a-e780-4ac5-aa78-0e74057526eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:46:50.547480 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.547462 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ae41d9a-e780-4ac5-aa78-0e74057526eb" (UID: "4ae41d9a-e780-4ac5-aa78-0e74057526eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:46:50.646381 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.646345 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:46:50.646381 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.646375 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgvv\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-kube-api-access-ncgvv\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:46:50.646381 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.646390 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:46:50.646826 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.646401 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:46:50.646826 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.646414 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:46:50.646826 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.646424 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae41d9a-e780-4ac5-aa78-0e74057526eb-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:46:50.646826 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:50.646435 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae41d9a-e780-4ac5-aa78-0e74057526eb-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:46:51.034150 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.034109 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerID="247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f" exitCode=0 Jul 14 22:46:51.034344 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.034158 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" Jul 14 22:46:51.034344 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.034186 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" event={"ID":"4ae41d9a-e780-4ac5-aa78-0e74057526eb","Type":"ContainerDied","Data":"247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f"} Jul 14 22:46:51.034344 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.034262 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff" event={"ID":"4ae41d9a-e780-4ac5-aa78-0e74057526eb","Type":"ContainerDied","Data":"34309ac96d5755d03e6e9815592f2e53281eba0c5728b9952053c31a844a971f"} Jul 14 22:46:51.034344 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.034291 2578 scope.go:117] "RemoveContainer" containerID="247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f" Jul 14 22:46:51.043802 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.043788 2578 scope.go:117] "RemoveContainer" containerID="88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5" Jul 14 22:46:51.053445 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.053419 2578 scope.go:117] "RemoveContainer" containerID="247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f" Jul 14 22:46:51.053809 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:46:51.053781 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f\": container with ID starting with 247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f not found: ID does not exist" containerID="247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f" Jul 14 22:46:51.053911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.053819 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f"} err="failed to get container status \"247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f\": rpc error: code = NotFound desc = could not find container \"247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f\": container with ID starting with 247a3d0aab88fdff9c4733c3bf7f6ac4088d1eb7dac60f5e2df3914f84944e9f not found: ID does not exist" Jul 14 22:46:51.053911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.053837 2578 scope.go:117] "RemoveContainer" containerID="88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5" Jul 14 22:46:51.054163 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:46:51.054123 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5\": container with ID starting with 88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5 not found: ID does not exist" containerID="88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5" Jul 14 22:46:51.054303 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.054161 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5"} err="failed to get container status \"88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5\": rpc error: code = NotFound desc = could not find container \"88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5\": container with ID starting with 88240aeee15cb4a07f519fa9617cb717220ad25719a208e16b052501d6be04a5 not found: ID does not exist" Jul 14 22:46:51.057608 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.057585 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff"] Jul 14 22:46:51.060888 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:51.060869 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7a211570-openshift-default-d8955b6d7-stwff"] Jul 14 22:46:52.839395 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:46:52.839353 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" path="/var/lib/kubelet/pods/4ae41d9a-e780-4ac5-aa78-0e74057526eb/volumes" Jul 14 22:47:03.402005 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:03.401964 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:47:42.146359 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.146322 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns"] Jul 14 22:47:42.146824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.146745 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" Jul 14 22:47:42.146824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.146760 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" Jul 14 22:47:42.146824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.146779 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c838f78f-ff88-4e13-b162-ed646c8665c5" containerName="revoke-keys" Jul 14 22:47:42.146824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.146785 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c838f78f-ff88-4e13-b162-ed646c8665c5" containerName="revoke-keys" Jul 14 22:47:42.146962 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.146856 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" Jul 14 22:47:42.146962 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.146870 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c838f78f-ff88-4e13-b162-ed646c8665c5" containerName="revoke-keys" Jul 14 22:47:42.149844 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.149828 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.153157 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.153135 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fcf58c-openshift-default-dockercfg-btvcx\"" Jul 14 22:47:42.170442 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.170420 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns"] Jul 14 22:47:42.175392 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.175363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-data\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.175502 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.175394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-token\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.175502 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.175417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-podinfo\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.175502 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.175460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-credential-socket\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.175502 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.175477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-certs\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.175502 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.175491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istiod-ca-cert\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.175772 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.175507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdn87\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-kube-api-access-xdn87\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.175772 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.175592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-envoy\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.175772 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.175672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-socket\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.276824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.276788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-credential-socket\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.276824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.276822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-certs\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.276837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istiod-ca-cert\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.276857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdn87\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-kube-api-access-xdn87\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-envoy\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277252 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-socket\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277252 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-data\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277373 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-token\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277373 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-podinfo\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277373 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277327 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-credential-socket\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277373 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277252 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-certs\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277640 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277502 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-data\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277640 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277558 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-socket\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.277834 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.277812 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istiod-ca-cert\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.279435 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.279414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-envoy\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.279781 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.279763 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-podinfo\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.285158 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.285134 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-token\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.285285 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.285264 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdn87\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-kube-api-access-xdn87\") pod \"e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.461232 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.461135 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:42.598520 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.598495 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns"] Jul 14 22:47:42.600162 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:47:42.600107 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63bf5b0_b210_4aa4_89f8_4e235630e56a.slice/crio-95c61a1606c563e4a67e5d6bf3d62bfb660b13ff6f607868f80fc745796dfab0 WatchSource:0}: Error finding container 95c61a1606c563e4a67e5d6bf3d62bfb660b13ff6f607868f80fc745796dfab0: Status 404 returned error can't find the container with id 95c61a1606c563e4a67e5d6bf3d62bfb660b13ff6f607868f80fc745796dfab0 Jul 14 22:47:42.602527 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.602484 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:47:42.602610 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.602577 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:47:42.602646 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:42.602621 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:47:43.307474 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:43.307444 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" event={"ID":"b63bf5b0-b210-4aa4-89f8-4e235630e56a","Type":"ContainerStarted","Data":"92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2"} Jul 14 22:47:43.307838 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:43.307491 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" event={"ID":"b63bf5b0-b210-4aa4-89f8-4e235630e56a","Type":"ContainerStarted","Data":"95c61a1606c563e4a67e5d6bf3d62bfb660b13ff6f607868f80fc745796dfab0"} Jul 14 22:47:43.332955 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:43.332907 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" podStartSLOduration=1.332892097 podStartE2EDuration="1.332892097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:47:43.329840221 +0000 UTC m=+2329.142471874" watchObservedRunningTime="2026-07-14 22:47:43.332892097 +0000 UTC m=+2329.145523737" Jul 14 22:47:43.461988 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:43.461955 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:43.466740 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:43.466716 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:44.313075 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:44.313045 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:44.314045 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:44.314024 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:47:50.464575 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.464540 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g"] Jul 14 22:47:50.465097 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.465082 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" Jul 14 22:47:50.465146 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.465099 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" Jul 14 22:47:50.465225 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.465215 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae41d9a-e780-4ac5-aa78-0e74057526eb" containerName="istio-proxy" Jul 14 22:47:50.468983 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.468967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.473125 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.473103 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cb7974-openshift-default-dockercfg-6d622\"" Jul 14 22:47:50.488792 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.488769 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g"] Jul 14 22:47:50.548334 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.548301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-podinfo\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.548334 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.548347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-data\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.548609 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.548373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-certs\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.548609 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.548395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-credential-socket\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.548609 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.548512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-envoy\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.548609 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.548557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-token\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.548821 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.548609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istiod-ca-cert\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.548821 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.548640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-socket\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.548821 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.548668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxm8f\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-kube-api-access-gxm8f\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.649595 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.649563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-envoy\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.649595 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.649604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-token\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.649877 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.649638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istiod-ca-cert\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.649877 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.649668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-socket\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.649877 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.649823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxm8f\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-kube-api-access-gxm8f\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.650036 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.649948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-podinfo\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.650036 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.649978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-data\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.650036 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.650007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-certs\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.650193 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.650036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-credential-socket\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.650193 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.650071 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-socket\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.650379 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.650340 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-data\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.650502 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.650399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-credential-socket\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.650502 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.650426 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-certs\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.650582 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.650546 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istiod-ca-cert\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.652038 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.652005 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-envoy\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.652397 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.652375 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-podinfo\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.664314 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.664289 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-token\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.664680 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.664657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxm8f\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-kube-api-access-gxm8f\") pod \"e2e-inf-b-cb7974-openshift-default-744d896995-h488g\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.781235 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.781145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:50.929839 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.929813 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g"] Jul 14 22:47:50.931222 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:47:50.931199 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4418b4a8_3486_49a1_926d_aa5ba0ae403c.slice/crio-84ee63ee973e3126f089476baadddfa7dbb3affad58f80a709676d4b087a2cc9 WatchSource:0}: Error finding container 84ee63ee973e3126f089476baadddfa7dbb3affad58f80a709676d4b087a2cc9: Status 404 returned error can't find the container with id 84ee63ee973e3126f089476baadddfa7dbb3affad58f80a709676d4b087a2cc9 Jul 14 22:47:50.933483 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.933450 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:47:50.933577 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.933515 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:47:50.933577 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:50.933544 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:47:51.352731 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:51.352690 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" event={"ID":"4418b4a8-3486-49a1-926d-aa5ba0ae403c","Type":"ContainerStarted","Data":"86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9"} Jul 14 22:47:51.352902 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:51.352741 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" event={"ID":"4418b4a8-3486-49a1-926d-aa5ba0ae403c","Type":"ContainerStarted","Data":"84ee63ee973e3126f089476baadddfa7dbb3affad58f80a709676d4b087a2cc9"} Jul 14 22:47:51.426328 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:51.426271 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" podStartSLOduration=1.4262347960000001 podStartE2EDuration="1.426234796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:47:51.421368822 +0000 UTC m=+2337.234000461" watchObservedRunningTime="2026-07-14 22:47:51.426234796 +0000 UTC m=+2337.238866437" Jul 14 22:47:51.782176 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:51.782081 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:51.787146 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:51.787121 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:52.358604 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:52.358574 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:52.359589 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:52.359564 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:47:53.305122 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:53.305089 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:47:54.964812 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:47:54.964773 2578 scope.go:117] "RemoveContainer" containerID="dfcfeb664a43926a50c77ef5f12188fc0be583fa01d1e7f690ffe39d4e5d966f" Jul 14 22:48:00.355423 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:00.355383 2578 patch_prober.go:28] interesting pod/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 22:48:00.355904 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:00.355457 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 22:48:01.414845 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:01.414765 2578 generic.go:356] "Generic (PLEG): container finished" podID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerID="92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2" exitCode=0 Jul 14 22:48:01.415218 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:01.414842 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" event={"ID":"b63bf5b0-b210-4aa4-89f8-4e235630e56a","Type":"ContainerDied","Data":"92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2"} Jul 14 22:48:01.415218 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:01.415214 2578 scope.go:117] "RemoveContainer" containerID="92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2" Jul 14 22:48:01.416252 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:01.416210 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:48:01.416363 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:01.416291 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:48:01.416363 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:01.416324 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:48:02.422419 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:02.422382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" event={"ID":"b63bf5b0-b210-4aa4-89f8-4e235630e56a","Type":"ContainerStarted","Data":"44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207"} Jul 14 22:48:02.462705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:02.462660 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:48:02.463119 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:02.463093 2578 patch_prober.go:28] interesting pod/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 14 22:48:02.463332 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:02.463305 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 14 22:48:03.462342 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:03.462303 2578 patch_prober.go:28] interesting pod/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 14 22:48:03.462943 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:03.462910 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 14 22:48:04.462429 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:04.462372 2578 patch_prober.go:28] interesting pod/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 14 22:48:04.462941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:04.462455 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 14 22:48:05.462286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:05.462223 2578 patch_prober.go:28] interesting pod/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 14 22:48:05.462602 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:05.462571 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 14 22:48:06.465065 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:06.465035 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:48:06.465510 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:06.465489 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:48:06.466329 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:06.466311 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:48:07.590492 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:07.590450 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:48:08.400407 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:08.400373 2578 patch_prober.go:28] interesting pod/e2e-inf-b-cb7974-openshift-default-744d896995-h488g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 22:48:08.400588 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:08.400464 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 22:48:09.467382 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:09.467277 2578 generic.go:356] "Generic (PLEG): container finished" podID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerID="86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9" exitCode=0 Jul 14 22:48:09.467382 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:09.467358 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" event={"ID":"4418b4a8-3486-49a1-926d-aa5ba0ae403c","Type":"ContainerDied","Data":"86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9"} Jul 14 22:48:09.467820 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:09.467732 2578 scope.go:117] "RemoveContainer" containerID="86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9" Jul 14 22:48:09.468747 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:09.468717 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:48:09.468854 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:09.468809 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:48:09.468925 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:09.468863 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:48:10.474981 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:10.474947 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" event={"ID":"4418b4a8-3486-49a1-926d-aa5ba0ae403c","Type":"ContainerStarted","Data":"6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e"} Jul 14 22:48:10.781613 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:10.781515 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:48:10.781785 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:10.781763 2578 patch_prober.go:28] interesting pod/e2e-inf-b-cb7974-openshift-default-744d896995-h488g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 14 22:48:10.781831 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:10.781810 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 14 22:48:11.782193 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:11.782156 2578 patch_prober.go:28] interesting pod/e2e-inf-b-cb7974-openshift-default-744d896995-h488g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 14 22:48:11.782624 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:11.782222 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 14 22:48:12.782116 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:12.782076 2578 patch_prober.go:28] interesting pod/e2e-inf-b-cb7974-openshift-default-744d896995-h488g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 14 22:48:12.782348 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:12.782135 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 14 22:48:13.782401 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:13.782365 2578 patch_prober.go:28] interesting pod/e2e-inf-b-cb7974-openshift-default-744d896995-h488g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 14 22:48:13.782806 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:13.782425 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 14 22:48:14.785116 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:14.785081 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:48:14.785555 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:14.785472 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:48:14.786104 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:14.786087 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:48:16.891752 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:16.891720 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:48:30.790651 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:30.790616 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:48:45.094124 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:45.094088 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:48:54.848352 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:54.848321 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:48:54.850633 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:54.850612 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:48:58.384296 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:48:58.384262 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:49:13.060070 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.060031 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g"] Jul 14 22:49:13.063816 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.063797 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" Jul 14 22:49:13.066979 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.066957 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:49:13.066979 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.066971 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:49:13.071508 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.071480 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g"] Jul 14 22:49:13.223716 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.223680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22849bec-7c52-4ec0-8160-67b8fdd67abf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g\" (UID: \"22849bec-7c52-4ec0-8160-67b8fdd67abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" Jul 14 22:49:13.324779 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.324692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22849bec-7c52-4ec0-8160-67b8fdd67abf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g\" (UID: \"22849bec-7c52-4ec0-8160-67b8fdd67abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" Jul 14 22:49:13.325353 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.325329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22849bec-7c52-4ec0-8160-67b8fdd67abf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g\" (UID: \"22849bec-7c52-4ec0-8160-67b8fdd67abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" Jul 14 22:49:13.374700 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.374667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" Jul 14 22:49:13.512854 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.512830 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g"] Jul 14 22:49:13.514874 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:49:13.514847 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22849bec_7c52_4ec0_8160_67b8fdd67abf.slice/crio-a47a1085c8ca3c4917bfc3e19c9a48f986758501878fd1ce1e39e0c15f644d01 WatchSource:0}: Error finding container a47a1085c8ca3c4917bfc3e19c9a48f986758501878fd1ce1e39e0c15f644d01: Status 404 returned error can't find the container with id a47a1085c8ca3c4917bfc3e19c9a48f986758501878fd1ce1e39e0c15f644d01 Jul 14 22:49:13.817663 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.817628 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" event={"ID":"22849bec-7c52-4ec0-8160-67b8fdd67abf","Type":"ContainerStarted","Data":"24f806ed6706367a15283364843450978c2589762735f3706506d56d7d35c459"} Jul 14 22:49:13.817825 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.817683 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" event={"ID":"22849bec-7c52-4ec0-8160-67b8fdd67abf","Type":"ContainerStarted","Data":"a47a1085c8ca3c4917bfc3e19c9a48f986758501878fd1ce1e39e0c15f644d01"} Jul 14 22:49:13.834287 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:13.834216 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" podStartSLOduration=0.83419692 podStartE2EDuration="834.19692ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:49:13.832496457 +0000 UTC m=+2419.645128100" watchObservedRunningTime="2026-07-14 22:49:13.83419692 +0000 UTC m=+2419.646828561" Jul 14 22:49:14.827395 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:14.827352 2578 generic.go:356] "Generic (PLEG): container finished" podID="22849bec-7c52-4ec0-8160-67b8fdd67abf" containerID="24f806ed6706367a15283364843450978c2589762735f3706506d56d7d35c459" exitCode=0 Jul 14 22:49:14.827774 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:14.827433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" event={"ID":"22849bec-7c52-4ec0-8160-67b8fdd67abf","Type":"ContainerDied","Data":"24f806ed6706367a15283364843450978c2589762735f3706506d56d7d35c459"} Jul 14 22:49:15.973924 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:15.973899 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" Jul 14 22:49:16.151621 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:16.151535 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22849bec-7c52-4ec0-8160-67b8fdd67abf-maas-api-service-ca\") pod \"22849bec-7c52-4ec0-8160-67b8fdd67abf\" (UID: \"22849bec-7c52-4ec0-8160-67b8fdd67abf\") " Jul 14 22:49:16.151941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:16.151911 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22849bec-7c52-4ec0-8160-67b8fdd67abf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22849bec-7c52-4ec0-8160-67b8fdd67abf" (UID: "22849bec-7c52-4ec0-8160-67b8fdd67abf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:49:16.196928 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:16.196897 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:49:16.252497 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:16.252468 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22849bec-7c52-4ec0-8160-67b8fdd67abf-maas-api-service-ca\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:16.840638 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:16.840588 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" event={"ID":"22849bec-7c52-4ec0-8160-67b8fdd67abf","Type":"ContainerDied","Data":"a47a1085c8ca3c4917bfc3e19c9a48f986758501878fd1ce1e39e0c15f644d01"} Jul 14 22:49:16.840638 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:16.840644 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a47a1085c8ca3c4917bfc3e19c9a48f986758501878fd1ce1e39e0c15f644d01" Jul 14 22:49:16.840850 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:16.840611 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g" Jul 14 22:49:33.698429 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:33.698388 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:49:38.886856 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:38.886821 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns"] Jul 14 22:49:38.887372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:38.887077 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" containerID="cri-o://44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207" gracePeriod=30 Jul 14 22:49:44.149972 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.149946 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:49:44.201479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201395 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-credential-socket\") pod \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " Jul 14 22:49:44.201479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201431 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-envoy\") pod \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " Jul 14 22:49:44.201479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201455 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istiod-ca-cert\") pod \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " Jul 14 22:49:44.201479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201476 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-data\") pod \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " Jul 14 22:49:44.201801 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201570 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-podinfo\") pod \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " Jul 14 22:49:44.201801 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201618 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdn87\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-kube-api-access-xdn87\") pod \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " Jul 14 22:49:44.201801 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201640 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-socket\") pod \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " Jul 14 22:49:44.201801 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201682 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-token\") pod \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " Jul 14 22:49:44.202372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201693 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b63bf5b0-b210-4aa4-89f8-4e235630e56a" (UID: "b63bf5b0-b210-4aa4-89f8-4e235630e56a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:49:44.202372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201855 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b63bf5b0-b210-4aa4-89f8-4e235630e56a" (UID: "b63bf5b0-b210-4aa4-89f8-4e235630e56a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:49:44.202372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201725 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-certs\") pod \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\" (UID: \"b63bf5b0-b210-4aa4-89f8-4e235630e56a\") " Jul 14 22:49:44.202372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201936 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b63bf5b0-b210-4aa4-89f8-4e235630e56a" (UID: "b63bf5b0-b210-4aa4-89f8-4e235630e56a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:49:44.202372 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.201949 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b63bf5b0-b210-4aa4-89f8-4e235630e56a" (UID: "b63bf5b0-b210-4aa4-89f8-4e235630e56a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:49:44.202658 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.202500 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:44.202658 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.202520 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:44.202658 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.202536 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:44.202658 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.202549 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:44.202658 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.202569 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-data" (OuterVolumeSpecName: "istio-data") pod "b63bf5b0-b210-4aa4-89f8-4e235630e56a" (UID: "b63bf5b0-b210-4aa4-89f8-4e235630e56a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:49:44.204164 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.204009 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b63bf5b0-b210-4aa4-89f8-4e235630e56a" (UID: "b63bf5b0-b210-4aa4-89f8-4e235630e56a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:49:44.204164 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.204117 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b63bf5b0-b210-4aa4-89f8-4e235630e56a" (UID: "b63bf5b0-b210-4aa4-89f8-4e235630e56a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:49:44.204362 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.204172 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b63bf5b0-b210-4aa4-89f8-4e235630e56a" (UID: "b63bf5b0-b210-4aa4-89f8-4e235630e56a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:49:44.204362 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.204174 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-kube-api-access-xdn87" (OuterVolumeSpecName: "kube-api-access-xdn87") pod "b63bf5b0-b210-4aa4-89f8-4e235630e56a" (UID: "b63bf5b0-b210-4aa4-89f8-4e235630e56a"). InnerVolumeSpecName "kube-api-access-xdn87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:49:44.303233 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.303197 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:44.303233 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.303226 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:44.303233 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.303259 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:44.303476 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.303273 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdn87\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-kube-api-access-xdn87\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:44.303476 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.303286 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63bf5b0-b210-4aa4-89f8-4e235630e56a-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:49:44.996442 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.996409 2578 generic.go:356] "Generic (PLEG): container finished" podID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerID="44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207" exitCode=0 Jul 14 22:49:44.996603 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.996463 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" Jul 14 22:49:44.996603 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.996494 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" event={"ID":"b63bf5b0-b210-4aa4-89f8-4e235630e56a","Type":"ContainerDied","Data":"44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207"} Jul 14 22:49:44.996603 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.996555 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns" event={"ID":"b63bf5b0-b210-4aa4-89f8-4e235630e56a","Type":"ContainerDied","Data":"95c61a1606c563e4a67e5d6bf3d62bfb660b13ff6f607868f80fc745796dfab0"} Jul 14 22:49:44.996603 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:44.996586 2578 scope.go:117] "RemoveContainer" containerID="44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207" Jul 14 22:49:45.006308 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:45.006039 2578 scope.go:117] "RemoveContainer" containerID="92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2" Jul 14 22:49:45.016188 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:45.016167 2578 scope.go:117] "RemoveContainer" containerID="44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207" Jul 14 22:49:45.016537 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:49:45.016508 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207\": container with ID starting with 44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207 not found: ID does not exist" containerID="44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207" Jul 14 22:49:45.016640 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:45.016536 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207"} err="failed to get container status \"44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207\": rpc error: code = NotFound desc = could not find container \"44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207\": container with ID starting with 44a711617e1dc6e9fe818dfb4e28dc4ab5e00e8be574b88d0eb32583474d0207 not found: ID does not exist" Jul 14 22:49:45.016640 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:45.016553 2578 scope.go:117] "RemoveContainer" containerID="92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2" Jul 14 22:49:45.016876 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:49:45.016855 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2\": container with ID starting with 92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2 not found: ID does not exist" containerID="92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2" Jul 14 22:49:45.016941 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:45.016886 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2"} err="failed to get container status \"92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2\": rpc error: code = NotFound desc = could not find container \"92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2\": container with ID starting with 92e57b96c7542761b97ecbb7c4aae76148fc80a8d18c4c31c880b5fd86c5d4c2 not found: ID does not exist" Jul 14 22:49:45.018340 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:45.018318 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns"] Jul 14 22:49:45.021685 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:45.021662 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fcf58c-openshift-default-dcf4c567f-mvtns"] Jul 14 22:49:46.837534 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:46.837499 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" path="/var/lib/kubelet/pods/b63bf5b0-b210-4aa4-89f8-4e235630e56a/volumes" Jul 14 22:49:52.808195 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:49:52.808157 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:50:05.111629 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:05.111597 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g"] Jul 14 22:50:05.112092 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:05.111840 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" containerID="cri-o://6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e" gracePeriod=30 Jul 14 22:50:06.083322 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.083289 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp"] Jul 14 22:50:06.083746 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.083729 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" Jul 14 22:50:06.083837 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.083748 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" Jul 14 22:50:06.083837 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.083765 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22849bec-7c52-4ec0-8160-67b8fdd67abf" containerName="revoke-keys" Jul 14 22:50:06.083837 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.083773 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22849bec-7c52-4ec0-8160-67b8fdd67abf" containerName="revoke-keys" Jul 14 22:50:06.083995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.083896 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22849bec-7c52-4ec0-8160-67b8fdd67abf" containerName="revoke-keys" Jul 14 22:50:06.083995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.083913 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" Jul 14 22:50:06.083995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.083928 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" Jul 14 22:50:06.087131 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.087110 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.090670 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.090650 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-971ed6-openshift-default-dockercfg-w2xnh\"" Jul 14 22:50:06.103163 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.103137 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp"] Jul 14 22:50:06.194777 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.194747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-envoy\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.195268 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.194789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-credential-socket\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.195268 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.194825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-data\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.195268 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.194853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vls7\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-kube-api-access-8vls7\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.195268 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.194925 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istiod-ca-cert\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.195268 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.194978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-token\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.195268 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.195011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-socket\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.195268 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.195045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-certs\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.195268 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.195083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-podinfo\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.295856 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.295817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istiod-ca-cert\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.295871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-token\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.295893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-socket\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.295915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-certs\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.295938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-podinfo\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.295975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-envoy\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.295989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-credential-socket\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.296022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-data\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.296049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vls7\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-kube-api-access-8vls7\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.296378 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-socket\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296581 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.296472 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-certs\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296640 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.296617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-data\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296640 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.296630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istiod-ca-cert\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.296862 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.296843 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-credential-socket\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.298384 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.298363 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-envoy\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.298614 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.298593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-podinfo\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.305417 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.305390 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vls7\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-kube-api-access-8vls7\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.305832 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.305811 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-token\") pod \"e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.397092 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.397001 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:06.545973 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.545938 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp"] Jul 14 22:50:06.549917 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:50:06.549890 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbad1ca_7494_4295_aa36_7bb877ae2e51.slice/crio-ede6b8248af4591b6de291b711df4acf0edcde83422f916abf729c2f95356b9c WatchSource:0}: Error finding container ede6b8248af4591b6de291b711df4acf0edcde83422f916abf729c2f95356b9c: Status 404 returned error can't find the container with id ede6b8248af4591b6de291b711df4acf0edcde83422f916abf729c2f95356b9c Jul 14 22:50:06.552135 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.552095 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:06.552210 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.552186 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:06.552282 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:06.552232 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:07.119291 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:07.119247 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" event={"ID":"cdbad1ca-7494-4295-aa36-7bb877ae2e51","Type":"ContainerStarted","Data":"757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97"} Jul 14 22:50:07.119503 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:07.119311 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" event={"ID":"cdbad1ca-7494-4295-aa36-7bb877ae2e51","Type":"ContainerStarted","Data":"ede6b8248af4591b6de291b711df4acf0edcde83422f916abf729c2f95356b9c"} Jul 14 22:50:07.167255 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:07.167194 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" podStartSLOduration=1.167178748 podStartE2EDuration="1.167178748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:50:07.164136306 +0000 UTC m=+2472.976767945" watchObservedRunningTime="2026-07-14 22:50:07.167178748 +0000 UTC m=+2472.979810388" Jul 14 22:50:07.398077 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:07.397981 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:07.403031 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:07.403007 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:08.126104 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:08.126072 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:08.127040 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:08.127023 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:10.370084 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.370055 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:50:10.432913 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.432820 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-socket\") pod \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " Jul 14 22:50:10.432913 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.432864 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-envoy\") pod \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " Jul 14 22:50:10.432913 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.432896 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-credential-socket\") pod \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " Jul 14 22:50:10.433189 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.432924 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-podinfo\") pod \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " Jul 14 22:50:10.433189 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.432968 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-token\") pod \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " Jul 14 22:50:10.433189 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.432999 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-certs\") pod \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " Jul 14 22:50:10.433189 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.433051 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxm8f\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-kube-api-access-gxm8f\") pod \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " Jul 14 22:50:10.433189 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.433089 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-data\") pod \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " Jul 14 22:50:10.433189 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.433117 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istiod-ca-cert\") pod \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\" (UID: \"4418b4a8-3486-49a1-926d-aa5ba0ae403c\") " Jul 14 22:50:10.433189 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.433153 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4418b4a8-3486-49a1-926d-aa5ba0ae403c" (UID: "4418b4a8-3486-49a1-926d-aa5ba0ae403c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:50:10.433579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.433285 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4418b4a8-3486-49a1-926d-aa5ba0ae403c" (UID: "4418b4a8-3486-49a1-926d-aa5ba0ae403c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:50:10.433579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.433399 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4418b4a8-3486-49a1-926d-aa5ba0ae403c" (UID: "4418b4a8-3486-49a1-926d-aa5ba0ae403c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:50:10.433579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.433492 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:50:10.433579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.433510 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:50:10.433579 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.433525 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:50:10.434448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.434299 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-data" (OuterVolumeSpecName: "istio-data") pod "4418b4a8-3486-49a1-926d-aa5ba0ae403c" (UID: "4418b4a8-3486-49a1-926d-aa5ba0ae403c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:50:10.434448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.434378 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4418b4a8-3486-49a1-926d-aa5ba0ae403c" (UID: "4418b4a8-3486-49a1-926d-aa5ba0ae403c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:50:10.435570 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.435543 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4418b4a8-3486-49a1-926d-aa5ba0ae403c" (UID: "4418b4a8-3486-49a1-926d-aa5ba0ae403c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:50:10.435712 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.435593 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4418b4a8-3486-49a1-926d-aa5ba0ae403c" (UID: "4418b4a8-3486-49a1-926d-aa5ba0ae403c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:50:10.436092 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.436065 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-token" (OuterVolumeSpecName: "istio-token") pod "4418b4a8-3486-49a1-926d-aa5ba0ae403c" (UID: "4418b4a8-3486-49a1-926d-aa5ba0ae403c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:50:10.436216 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.436118 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-kube-api-access-gxm8f" (OuterVolumeSpecName: "kube-api-access-gxm8f") pod "4418b4a8-3486-49a1-926d-aa5ba0ae403c" (UID: "4418b4a8-3486-49a1-926d-aa5ba0ae403c"). InnerVolumeSpecName "kube-api-access-gxm8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:50:10.534104 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.534073 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:50:10.534104 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.534101 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:50:10.534104 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.534110 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:50:10.534363 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.534118 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxm8f\" (UniqueName: \"kubernetes.io/projected/4418b4a8-3486-49a1-926d-aa5ba0ae403c-kube-api-access-gxm8f\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:50:10.534363 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.534130 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:50:10.534363 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:10.534142 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4418b4a8-3486-49a1-926d-aa5ba0ae403c-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:50:11.146936 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.146828 2578 generic.go:356] "Generic (PLEG): container finished" podID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerID="6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e" exitCode=0 Jul 14 22:50:11.146936 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.146876 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" Jul 14 22:50:11.146936 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.146919 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" event={"ID":"4418b4a8-3486-49a1-926d-aa5ba0ae403c","Type":"ContainerDied","Data":"6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e"} Jul 14 22:50:11.147198 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.146977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g" event={"ID":"4418b4a8-3486-49a1-926d-aa5ba0ae403c","Type":"ContainerDied","Data":"84ee63ee973e3126f089476baadddfa7dbb3affad58f80a709676d4b087a2cc9"} Jul 14 22:50:11.147198 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.147002 2578 scope.go:117] "RemoveContainer" containerID="6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e" Jul 14 22:50:11.155851 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.155832 2578 scope.go:117] "RemoveContainer" containerID="86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9" Jul 14 22:50:11.163801 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.163783 2578 scope.go:117] "RemoveContainer" containerID="6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e" Jul 14 22:50:11.164022 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:50:11.164003 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e\": container with ID starting with 6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e not found: ID does not exist" containerID="6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e" Jul 14 22:50:11.164074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.164032 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e"} err="failed to get container status \"6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e\": rpc error: code = NotFound desc = could not find container \"6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e\": container with ID starting with 6e3677d74df097f8cf250642d6d35a17906220c7a1f464e9c11dba143382930e not found: ID does not exist" Jul 14 22:50:11.164074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.164050 2578 scope.go:117] "RemoveContainer" containerID="86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9" Jul 14 22:50:11.164314 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:50:11.164298 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9\": container with ID starting with 86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9 not found: ID does not exist" containerID="86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9" Jul 14 22:50:11.164374 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.164317 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9"} err="failed to get container status \"86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9\": rpc error: code = NotFound desc = could not find container \"86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9\": container with ID starting with 86507ccf7100c142e956ff505fbf9963501654ad3d9186b2293ecf2c12cc04b9 not found: ID does not exist" Jul 14 22:50:11.179701 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.179678 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g"] Jul 14 22:50:11.183947 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:11.183929 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cb7974-openshift-default-744d896995-h488g"] Jul 14 22:50:12.839657 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:12.839619 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" path="/var/lib/kubelet/pods/4418b4a8-3486-49a1-926d-aa5ba0ae403c/volumes" Jul 14 22:50:13.707607 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:13.707574 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:50:14.687992 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.687945 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp"] Jul 14 22:50:14.688474 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.688370 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" Jul 14 22:50:14.688474 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.688383 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" Jul 14 22:50:14.688474 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.688398 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" Jul 14 22:50:14.688474 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.688406 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63bf5b0-b210-4aa4-89f8-4e235630e56a" containerName="istio-proxy" Jul 14 22:50:14.688474 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.688414 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" Jul 14 22:50:14.688474 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.688420 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" Jul 14 22:50:14.688797 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.688493 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" Jul 14 22:50:14.688797 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.688501 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4418b4a8-3486-49a1-926d-aa5ba0ae403c" containerName="istio-proxy" Jul 14 22:50:14.693080 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.693059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.695937 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.695918 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7215d1-openshift-default-dockercfg-xgb8l\"" Jul 14 22:50:14.729064 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.729029 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp"] Jul 14 22:50:14.775264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.773736 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4ac872-d541-41f2-8a01-da6dd93e9969-istiod-ca-cert\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.775264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.773791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pvr5\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-kube-api-access-9pvr5\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.775264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.773839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-data\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.775264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.773886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-socket\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.775264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.773918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-envoy\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.775264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.773967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-certs\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.775264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.774005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-token\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.775264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.774035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-credential-socket\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.775264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.774096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-podinfo\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875370 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4ac872-d541-41f2-8a01-da6dd93e9969-istiod-ca-cert\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875581 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvr5\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-kube-api-access-9pvr5\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875581 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-data\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875581 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-socket\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875581 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-envoy\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875581 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-certs\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875581 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-token\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875917 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-credential-socket\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875917 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-podinfo\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.875917 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875826 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-data\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.876093 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.875913 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-socket\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.876194 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.876172 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4ac872-d541-41f2-8a01-da6dd93e9969-istiod-ca-cert\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.876309 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.876288 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-credential-socket\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.876385 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.876368 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-certs\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.878031 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.878012 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-envoy\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.878149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.878131 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-podinfo\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.884749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.884721 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvr5\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-kube-api-access-9pvr5\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:14.885352 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:14.885334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-token\") pod \"e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:15.004235 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:15.004149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:15.206689 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:15.206659 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp"] Jul 14 22:50:15.207477 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:50:15.207451 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4ac872_d541_41f2_8a01_da6dd93e9969.slice/crio-ecaade215f93c49697c1bd74b93e5146f396ccc459e021afa24c36711b4186b8 WatchSource:0}: Error finding container ecaade215f93c49697c1bd74b93e5146f396ccc459e021afa24c36711b4186b8: Status 404 returned error can't find the container with id ecaade215f93c49697c1bd74b93e5146f396ccc459e021afa24c36711b4186b8 Jul 14 22:50:15.209485 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:15.209452 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:15.209564 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:15.209517 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:15.209564 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:15.209546 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:16.178035 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:16.178000 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" event={"ID":"dd4ac872-d541-41f2-8a01-da6dd93e9969","Type":"ContainerStarted","Data":"00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89"} Jul 14 22:50:16.178467 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:16.178055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" event={"ID":"dd4ac872-d541-41f2-8a01-da6dd93e9969","Type":"ContainerStarted","Data":"ecaade215f93c49697c1bd74b93e5146f396ccc459e021afa24c36711b4186b8"} Jul 14 22:50:16.203787 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:16.203736 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" podStartSLOduration=2.203721562 podStartE2EDuration="2.203721562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:50:16.20040414 +0000 UTC m=+2482.013035780" watchObservedRunningTime="2026-07-14 22:50:16.203721562 +0000 UTC m=+2482.016353202" Jul 14 22:50:17.004630 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:17.004595 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:17.009421 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:17.009395 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:17.184383 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:17.184350 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:17.185352 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:17.185331 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:22.216988 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:22.216957 2578 generic.go:356] "Generic (PLEG): container finished" podID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerID="757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97" exitCode=0 Jul 14 22:50:22.217460 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:22.217029 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" event={"ID":"cdbad1ca-7494-4295-aa36-7bb877ae2e51","Type":"ContainerDied","Data":"757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97"} Jul 14 22:50:22.217460 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:22.217442 2578 scope.go:117] "RemoveContainer" containerID="757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97" Jul 14 22:50:22.218481 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:22.218453 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:22.218599 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:22.218539 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:22.218599 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:22.218585 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:23.126341 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:23.126307 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:23.223955 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:23.223924 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" event={"ID":"cdbad1ca-7494-4295-aa36-7bb877ae2e51","Type":"ContainerStarted","Data":"e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c"} Jul 14 22:50:23.397792 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:23.397698 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:23.398908 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:23.398885 2578 patch_prober.go:28] interesting pod/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 14 22:50:23.398968 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:23.398940 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 14 22:50:24.397830 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:24.397796 2578 patch_prober.go:28] interesting pod/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 14 22:50:24.398209 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:24.397846 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 14 22:50:24.620150 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:24.620116 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:50:25.398075 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:25.398040 2578 patch_prober.go:28] interesting pod/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 14 22:50:25.398504 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:25.398098 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 14 22:50:26.397765 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:26.397726 2578 patch_prober.go:28] interesting pod/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 14 22:50:26.397960 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:26.397780 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 14 22:50:27.401349 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:27.401314 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:27.401724 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:27.401578 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:27.402280 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:27.402261 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:50:32.854147 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:32.853012 2578 patch_prober.go:28] interesting pod/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": read tcp 10.132.0.2:37956->10.132.0.68:15021: read: connection reset by peer" start-of-body= Jul 14 22:50:32.854147 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:32.853073 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": read tcp 10.132.0.2:37956->10.132.0.68:15021: read: connection reset by peer" Jul 14 22:50:33.292659 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:33.292566 2578 generic.go:356] "Generic (PLEG): container finished" podID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerID="00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89" exitCode=0 Jul 14 22:50:33.292659 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:33.292635 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" event={"ID":"dd4ac872-d541-41f2-8a01-da6dd93e9969","Type":"ContainerDied","Data":"00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89"} Jul 14 22:50:33.293112 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:33.293095 2578 scope.go:117] "RemoveContainer" containerID="00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89" Jul 14 22:50:33.294090 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:33.294058 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:33.294151 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:33.294126 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:33.294194 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:33.294160 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:50:34.301220 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:34.301187 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" event={"ID":"dd4ac872-d541-41f2-8a01-da6dd93e9969","Type":"ContainerStarted","Data":"25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d"} Jul 14 22:50:35.004568 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:35.004533 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:35.004779 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:35.004760 2578 patch_prober.go:28] interesting pod/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 14 22:50:35.004828 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:35.004804 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 14 22:50:36.004650 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:36.004609 2578 patch_prober.go:28] interesting pod/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 14 22:50:36.005118 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:36.004701 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 14 22:50:37.005007 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:37.004972 2578 patch_prober.go:28] interesting pod/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 14 22:50:37.005431 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:37.005026 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 14 22:50:38.004758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:38.004722 2578 patch_prober.go:28] interesting pod/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 14 22:50:38.004942 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:38.004779 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 14 22:50:39.008233 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:39.008202 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:39.008631 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:39.008506 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:39.009136 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:39.009120 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:50:45.409892 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:45.409845 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:50:56.933818 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:56.933781 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp"] Jul 14 22:50:56.934511 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:56.934046 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" containerID="cri-o://e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c" gracePeriod=30 Jul 14 22:50:57.403877 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:57.403841 2578 patch_prober.go:28] interesting pod/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:50:57.404045 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:50:57.403908 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:51:02.186655 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.186631 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:51:02.225972 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.225896 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-credential-socket\") pod \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " Jul 14 22:51:02.225972 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.225938 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-data\") pod \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " Jul 14 22:51:02.225972 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.225955 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-certs\") pod \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " Jul 14 22:51:02.226264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.225979 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-envoy\") pod \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " Jul 14 22:51:02.226264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226015 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-socket\") pod \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " Jul 14 22:51:02.226264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226094 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-token\") pod \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " Jul 14 22:51:02.226264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226132 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-podinfo\") pod \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " Jul 14 22:51:02.226264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226163 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vls7\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-kube-api-access-8vls7\") pod \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " Jul 14 22:51:02.226264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226193 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istiod-ca-cert\") pod \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\" (UID: \"cdbad1ca-7494-4295-aa36-7bb877ae2e51\") " Jul 14 22:51:02.226264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226205 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cdbad1ca-7494-4295-aa36-7bb877ae2e51" (UID: "cdbad1ca-7494-4295-aa36-7bb877ae2e51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:02.226627 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226465 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cdbad1ca-7494-4295-aa36-7bb877ae2e51" (UID: "cdbad1ca-7494-4295-aa36-7bb877ae2e51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:02.226627 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226479 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cdbad1ca-7494-4295-aa36-7bb877ae2e51" (UID: "cdbad1ca-7494-4295-aa36-7bb877ae2e51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:02.226627 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226571 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:02.226627 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226590 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:02.226822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226739 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cdbad1ca-7494-4295-aa36-7bb877ae2e51" (UID: "cdbad1ca-7494-4295-aa36-7bb877ae2e51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:51:02.226822 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226748 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:02.227000 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.226977 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-data" (OuterVolumeSpecName: "istio-data") pod "cdbad1ca-7494-4295-aa36-7bb877ae2e51" (UID: "cdbad1ca-7494-4295-aa36-7bb877ae2e51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:02.228703 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.228676 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-token" (OuterVolumeSpecName: "istio-token") pod "cdbad1ca-7494-4295-aa36-7bb877ae2e51" (UID: "cdbad1ca-7494-4295-aa36-7bb877ae2e51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:51:02.228703 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.228688 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-kube-api-access-8vls7" (OuterVolumeSpecName: "kube-api-access-8vls7") pod "cdbad1ca-7494-4295-aa36-7bb877ae2e51" (UID: "cdbad1ca-7494-4295-aa36-7bb877ae2e51"). InnerVolumeSpecName "kube-api-access-8vls7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:51:02.228860 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.228844 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cdbad1ca-7494-4295-aa36-7bb877ae2e51" (UID: "cdbad1ca-7494-4295-aa36-7bb877ae2e51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:02.228959 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.228944 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cdbad1ca-7494-4295-aa36-7bb877ae2e51" (UID: "cdbad1ca-7494-4295-aa36-7bb877ae2e51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:51:02.327650 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.327611 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:02.327650 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.327642 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:02.327650 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.327653 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:02.327650 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.327660 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:02.327916 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.327669 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vls7\" (UniqueName: \"kubernetes.io/projected/cdbad1ca-7494-4295-aa36-7bb877ae2e51-kube-api-access-8vls7\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:02.327916 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.327678 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbad1ca-7494-4295-aa36-7bb877ae2e51-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:02.461064 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.461028 2578 generic.go:356] "Generic (PLEG): container finished" podID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerID="e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c" exitCode=0 Jul 14 22:51:02.461064 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.461068 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" event={"ID":"cdbad1ca-7494-4295-aa36-7bb877ae2e51","Type":"ContainerDied","Data":"e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c"} Jul 14 22:51:02.461323 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.461079 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" Jul 14 22:51:02.461323 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.461107 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp" event={"ID":"cdbad1ca-7494-4295-aa36-7bb877ae2e51","Type":"ContainerDied","Data":"ede6b8248af4591b6de291b711df4acf0edcde83422f916abf729c2f95356b9c"} Jul 14 22:51:02.461323 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.461129 2578 scope.go:117] "RemoveContainer" containerID="e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c" Jul 14 22:51:02.470688 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.470671 2578 scope.go:117] "RemoveContainer" containerID="757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97" Jul 14 22:51:02.478738 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.478722 2578 scope.go:117] "RemoveContainer" containerID="e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c" Jul 14 22:51:02.478998 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:51:02.478981 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c\": container with ID starting with e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c not found: ID does not exist" containerID="e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c" Jul 14 22:51:02.479041 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.479007 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c"} err="failed to get container status \"e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c\": rpc error: code = NotFound desc = could not find container \"e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c\": container with ID starting with e32b9ba7cdc73eb6da31dabcbd0d63663cb8da35485de021cdc8d6e5573de04c not found: ID does not exist" Jul 14 22:51:02.479041 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.479026 2578 scope.go:117] "RemoveContainer" containerID="757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97" Jul 14 22:51:02.479284 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:51:02.479262 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97\": container with ID starting with 757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97 not found: ID does not exist" containerID="757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97" Jul 14 22:51:02.479340 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.479290 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97"} err="failed to get container status \"757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97\": rpc error: code = NotFound desc = could not find container \"757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97\": container with ID starting with 757ce9c1624fe4cdb61ce1856386da29fe7102e5b33530387e0a147433f90f97 not found: ID does not exist" Jul 14 22:51:02.484440 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.484420 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp"] Jul 14 22:51:02.488818 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.488798 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-971ed6-openshift-default-6f7899d8d7-ws5xp"] Jul 14 22:51:02.838626 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:02.838594 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" path="/var/lib/kubelet/pods/cdbad1ca-7494-4295-aa36-7bb877ae2e51/volumes" Jul 14 22:51:10.983696 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:10.983660 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:51:23.145053 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:23.145017 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp"] Jul 14 22:51:23.145659 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:23.145299 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" containerID="cri-o://25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d" gracePeriod=30 Jul 14 22:51:24.010927 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.010888 2578 patch_prober.go:28] interesting pod/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:51:24.011111 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.010950 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:51:24.103565 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.103531 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv"] Jul 14 22:51:24.103948 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.103935 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" Jul 14 22:51:24.103997 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.103952 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" Jul 14 22:51:24.104047 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.104037 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" Jul 14 22:51:24.108584 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.108562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.111959 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.111937 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7ab9a6-openshift-default-dockercfg-fv79t\"" Jul 14 22:51:24.120964 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.120938 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv"] Jul 14 22:51:24.222116 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.222086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-envoy\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.222496 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.222131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-token\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.222496 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.222169 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-credential-socket\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.222496 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.222212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njfl\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-kube-api-access-4njfl\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.222496 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.222234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-podinfo\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.222496 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.222270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-data\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.222496 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.222289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdff0e37-5b13-4064-816b-ff1695b5acee-istiod-ca-cert\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.222496 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.222308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-socket\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.222496 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.222334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-certs\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.323602 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.323563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-credential-socket\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.323805 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.323616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njfl\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-kube-api-access-4njfl\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.323805 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.323640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-podinfo\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.323805 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.323666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-data\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.323805 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.323690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdff0e37-5b13-4064-816b-ff1695b5acee-istiod-ca-cert\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.323805 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.323708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-socket\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.324037 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.323988 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-credential-socket\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.324037 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.324019 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-socket\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.324149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.324041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-certs\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.324149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.324130 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-data\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.324149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.324141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-envoy\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.324352 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.324208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-token\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.324352 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.324276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-certs\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.324463 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.324397 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdff0e37-5b13-4064-816b-ff1695b5acee-istiod-ca-cert\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.326121 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.326098 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-envoy\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.326201 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.326146 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-podinfo\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.332794 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.332770 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njfl\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-kube-api-access-4njfl\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.332953 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.332934 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-token\") pod \"e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.419900 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.419867 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:24.561439 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.561411 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv"] Jul 14 22:51:24.563323 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:51:24.563291 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdff0e37_5b13_4064_816b_ff1695b5acee.slice/crio-60ffa0233e1bc26514bea164f8d4e1339431a03aaadd3f592f43280c73d8381c WatchSource:0}: Error finding container 60ffa0233e1bc26514bea164f8d4e1339431a03aaadd3f592f43280c73d8381c: Status 404 returned error can't find the container with id 60ffa0233e1bc26514bea164f8d4e1339431a03aaadd3f592f43280c73d8381c Jul 14 22:51:24.565464 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.565429 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:24.565558 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.565493 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:24.565558 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.565522 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:24.581308 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:24.581230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" event={"ID":"bdff0e37-5b13-4064-816b-ff1695b5acee","Type":"ContainerStarted","Data":"60ffa0233e1bc26514bea164f8d4e1339431a03aaadd3f592f43280c73d8381c"} Jul 14 22:51:25.588909 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:25.588875 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" event={"ID":"bdff0e37-5b13-4064-816b-ff1695b5acee","Type":"ContainerStarted","Data":"06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad"} Jul 14 22:51:25.613230 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:25.613182 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" podStartSLOduration=1.6131679989999999 podStartE2EDuration="1.613167999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:51:25.609927564 +0000 UTC m=+2551.422559204" watchObservedRunningTime="2026-07-14 22:51:25.613167999 +0000 UTC m=+2551.425799720" Jul 14 22:51:26.420568 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:26.420533 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:26.425373 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:26.425350 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:26.595226 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:26.595193 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:26.596170 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:26.596152 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:28.404558 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.404475 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:51:28.566885 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.566856 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-token\") pod \"dd4ac872-d541-41f2-8a01-da6dd93e9969\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " Jul 14 22:51:28.567058 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.566952 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-envoy\") pod \"dd4ac872-d541-41f2-8a01-da6dd93e9969\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " Jul 14 22:51:28.567058 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.566981 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-certs\") pod \"dd4ac872-d541-41f2-8a01-da6dd93e9969\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " Jul 14 22:51:28.567058 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567037 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pvr5\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-kube-api-access-9pvr5\") pod \"dd4ac872-d541-41f2-8a01-da6dd93e9969\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " Jul 14 22:51:28.567221 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567069 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-podinfo\") pod \"dd4ac872-d541-41f2-8a01-da6dd93e9969\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " Jul 14 22:51:28.567221 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567107 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-socket\") pod \"dd4ac872-d541-41f2-8a01-da6dd93e9969\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " Jul 14 22:51:28.567221 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4ac872-d541-41f2-8a01-da6dd93e9969-istiod-ca-cert\") pod \"dd4ac872-d541-41f2-8a01-da6dd93e9969\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " Jul 14 22:51:28.567221 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567186 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-credential-socket\") pod \"dd4ac872-d541-41f2-8a01-da6dd93e9969\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " Jul 14 22:51:28.567466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567226 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-data\") pod \"dd4ac872-d541-41f2-8a01-da6dd93e9969\" (UID: \"dd4ac872-d541-41f2-8a01-da6dd93e9969\") " Jul 14 22:51:28.567466 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567422 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd4ac872-d541-41f2-8a01-da6dd93e9969" (UID: "dd4ac872-d541-41f2-8a01-da6dd93e9969"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:28.567578 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567514 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd4ac872-d541-41f2-8a01-da6dd93e9969" (UID: "dd4ac872-d541-41f2-8a01-da6dd93e9969"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:28.567658 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567635 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd4ac872-d541-41f2-8a01-da6dd93e9969" (UID: "dd4ac872-d541-41f2-8a01-da6dd93e9969"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:28.567658 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567646 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:28.567796 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567665 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:28.567853 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.567807 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4ac872-d541-41f2-8a01-da6dd93e9969-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd4ac872-d541-41f2-8a01-da6dd93e9969" (UID: "dd4ac872-d541-41f2-8a01-da6dd93e9969"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:51:28.568420 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.568391 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd4ac872-d541-41f2-8a01-da6dd93e9969" (UID: "dd4ac872-d541-41f2-8a01-da6dd93e9969"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:28.569487 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.569462 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd4ac872-d541-41f2-8a01-da6dd93e9969" (UID: "dd4ac872-d541-41f2-8a01-da6dd93e9969"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:51:28.569582 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.569489 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd4ac872-d541-41f2-8a01-da6dd93e9969" (UID: "dd4ac872-d541-41f2-8a01-da6dd93e9969"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:51:28.569663 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.569648 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-kube-api-access-9pvr5" (OuterVolumeSpecName: "kube-api-access-9pvr5") pod "dd4ac872-d541-41f2-8a01-da6dd93e9969" (UID: "dd4ac872-d541-41f2-8a01-da6dd93e9969"). InnerVolumeSpecName "kube-api-access-9pvr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:51:28.569704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.569661 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd4ac872-d541-41f2-8a01-da6dd93e9969" (UID: "dd4ac872-d541-41f2-8a01-da6dd93e9969"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:51:28.606588 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.606564 2578 generic.go:356] "Generic (PLEG): container finished" podID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerID="25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d" exitCode=0 Jul 14 22:51:28.606704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.606615 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" Jul 14 22:51:28.606704 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.606658 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" event={"ID":"dd4ac872-d541-41f2-8a01-da6dd93e9969","Type":"ContainerDied","Data":"25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d"} Jul 14 22:51:28.606829 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.606725 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp" event={"ID":"dd4ac872-d541-41f2-8a01-da6dd93e9969","Type":"ContainerDied","Data":"ecaade215f93c49697c1bd74b93e5146f396ccc459e021afa24c36711b4186b8"} Jul 14 22:51:28.606829 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.606761 2578 scope.go:117] "RemoveContainer" containerID="25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d" Jul 14 22:51:28.616039 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.616023 2578 scope.go:117] "RemoveContainer" containerID="00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89" Jul 14 22:51:28.624013 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.623993 2578 scope.go:117] "RemoveContainer" containerID="25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d" Jul 14 22:51:28.624359 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:51:28.624336 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d\": container with ID starting with 25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d not found: ID does not exist" containerID="25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d" Jul 14 22:51:28.624450 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.624364 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d"} err="failed to get container status \"25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d\": rpc error: code = NotFound desc = could not find container \"25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d\": container with ID starting with 25d0f9bb9251733180e11753dd8c2e4b32604b9d5ba0f4bbc9b61d514de9310d not found: ID does not exist" Jul 14 22:51:28.624450 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.624379 2578 scope.go:117] "RemoveContainer" containerID="00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89" Jul 14 22:51:28.624614 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:51:28.624594 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89\": container with ID starting with 00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89 not found: ID does not exist" containerID="00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89" Jul 14 22:51:28.624674 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.624619 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89"} err="failed to get container status \"00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89\": rpc error: code = NotFound desc = could not find container \"00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89\": container with ID starting with 00a6dc2f30f696aa46919f9a2a3cc0a933f8602145cccfb105cb6601cd15db89 not found: ID does not exist" Jul 14 22:51:28.629669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.629650 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp"] Jul 14 22:51:28.635227 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.635206 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7215d1-openshift-default-54b6bc478f-xt7tp"] Jul 14 22:51:28.668815 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.668767 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:28.668815 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.668785 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pvr5\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-kube-api-access-9pvr5\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:28.668815 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.668795 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:28.668815 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.668803 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4ac872-d541-41f2-8a01-da6dd93e9969-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:28.668815 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.668811 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:28.668815 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.668819 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:28.669059 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.668827 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4ac872-d541-41f2-8a01-da6dd93e9969-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:51:28.838420 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:28.838391 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" path="/var/lib/kubelet/pods/dd4ac872-d541-41f2-8a01-da6dd93e9969/volumes" Jul 14 22:51:31.899753 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:31.899716 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:51:32.790686 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.790655 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65"] Jul 14 22:51:32.791122 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.791104 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" Jul 14 22:51:32.791217 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.791125 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" Jul 14 22:51:32.791217 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.791137 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" Jul 14 22:51:32.791217 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.791144 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" Jul 14 22:51:32.791399 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.791298 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" Jul 14 22:51:32.791399 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.791312 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" Jul 14 22:51:32.791399 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.791325 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbad1ca-7494-4295-aa36-7bb877ae2e51" containerName="istio-proxy" Jul 14 22:51:32.794547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.794528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:32.797793 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.797774 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-546296-openshift-default-dockercfg-bf7jh\"" Jul 14 22:51:32.809123 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.809100 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65"] Jul 14 22:51:32.906132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.906096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-credential-socket\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:32.906547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.906140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-socket\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:32.906547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.906164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-token\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:32.906547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.906187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-certs\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:32.906547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.906259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-data\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:32.906547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.906282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-envoy\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:32.906547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.906301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-podinfo\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:32.906547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.906328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istiod-ca-cert\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:32.906547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:32.906380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zrl\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-kube-api-access-h5zrl\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-data\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007479 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-envoy\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007739 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-podinfo\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007739 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istiod-ca-cert\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007739 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zrl\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-kube-api-access-h5zrl\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-credential-socket\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-socket\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-token\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007890 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-data\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.007911 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.007899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-certs\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.008175 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.008101 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-credential-socket\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.008235 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.008187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-socket\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.008235 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.008203 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-certs\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.008383 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.008295 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istiod-ca-cert\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.009786 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.009765 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-envoy\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.010190 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.010169 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-podinfo\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.016319 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.016298 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-token\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.016593 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.016572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zrl\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-kube-api-access-h5zrl\") pod \"e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.105057 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.105022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:33.298032 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.298004 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65"] Jul 14 22:51:33.300709 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:51:33.300681 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c78d76_cdc2_46b2_9cd1_5a79aa4257f6.slice/crio-55a1e1a27f90acc92d9c331145fe9df3eb63909c7a03119801664933f0d8fbbc WatchSource:0}: Error finding container 55a1e1a27f90acc92d9c331145fe9df3eb63909c7a03119801664933f0d8fbbc: Status 404 returned error can't find the container with id 55a1e1a27f90acc92d9c331145fe9df3eb63909c7a03119801664933f0d8fbbc Jul 14 22:51:33.302883 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.302847 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:33.302956 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.302913 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:33.302956 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.302942 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:33.639962 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.638905 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" event={"ID":"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6","Type":"ContainerStarted","Data":"95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9"} Jul 14 22:51:33.639962 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.638976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" event={"ID":"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6","Type":"ContainerStarted","Data":"55a1e1a27f90acc92d9c331145fe9df3eb63909c7a03119801664933f0d8fbbc"} Jul 14 22:51:33.672211 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:33.672151 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" podStartSLOduration=1.672133976 podStartE2EDuration="1.672133976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:51:33.670513129 +0000 UTC m=+2559.483144772" watchObservedRunningTime="2026-07-14 22:51:33.672133976 +0000 UTC m=+2559.484765616" Jul 14 22:51:34.106166 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:34.106125 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:34.111585 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:34.111560 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:34.647591 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:34.647555 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:34.655677 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:34.655653 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:39.679330 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:39.679301 2578 generic.go:356] "Generic (PLEG): container finished" podID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerID="06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad" exitCode=0 Jul 14 22:51:39.679763 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:39.679377 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" event={"ID":"bdff0e37-5b13-4064-816b-ff1695b5acee","Type":"ContainerDied","Data":"06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad"} Jul 14 22:51:39.679812 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:39.679803 2578 scope.go:117] "RemoveContainer" containerID="06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad" Jul 14 22:51:39.680893 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:39.680865 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:39.680966 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:39.680946 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:39.681010 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:39.680996 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:40.687413 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:40.687378 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" event={"ID":"bdff0e37-5b13-4064-816b-ff1695b5acee","Type":"ContainerStarted","Data":"d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7"} Jul 14 22:51:41.420963 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:41.420926 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:41.422007 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:41.421975 2578 patch_prober.go:28] interesting pod/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 14 22:51:41.422142 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:41.422037 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 14 22:51:41.595365 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:41.595317 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:42.421234 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:42.421196 2578 patch_prober.go:28] interesting pod/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 14 22:51:42.421659 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:42.421288 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 14 22:51:43.420931 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:43.420894 2578 patch_prober.go:28] interesting pod/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 14 22:51:43.421108 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:43.420947 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 14 22:51:44.420915 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:44.420881 2578 patch_prober.go:28] interesting pod/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 14 22:51:44.421332 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:44.420939 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 14 22:51:45.424962 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:45.424934 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:45.426084 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:45.426066 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:51:45.897390 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:45.897356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:51:53.768155 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:53.768122 2578 generic.go:356] "Generic (PLEG): container finished" podID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerID="95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9" exitCode=0 Jul 14 22:51:53.768605 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:53.768159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" event={"ID":"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6","Type":"ContainerDied","Data":"95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9"} Jul 14 22:51:53.768649 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:53.768635 2578 scope.go:117] "RemoveContainer" containerID="95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9" Jul 14 22:51:53.769636 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:53.769604 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:53.769692 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:53.769679 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:53.769729 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:53.769720 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:51:54.776414 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:54.776382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" event={"ID":"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6","Type":"ContainerStarted","Data":"c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd"} Jul 14 22:51:55.026173 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:55.026145 2578 scope.go:117] "RemoveContainer" containerID="b7ad04a1985f49b04facbc19c7bc8c25eaa3f3560399f433bf4c72ae1e06b9c7" Jul 14 22:51:55.105816 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:55.105795 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:55.105998 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:55.105975 2578 patch_prober.go:28] interesting pod/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 14 22:51:55.106063 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:55.106031 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 14 22:51:56.105809 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:56.105768 2578 patch_prober.go:28] interesting pod/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 14 22:51:56.106310 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:56.105841 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 14 22:51:57.106178 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:57.106141 2578 patch_prober.go:28] interesting pod/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 14 22:51:57.106592 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:57.106197 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 14 22:51:58.105930 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:58.105895 2578 patch_prober.go:28] interesting pod/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 14 22:51:58.106117 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:58.105950 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 14 22:51:59.109125 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:59.109093 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:59.109519 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:59.109344 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:51:59.109942 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:51:59.109924 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:52:13.091995 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:52:13.091961 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:52:43.094959 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:52:43.094878 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:52:59.988389 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:52:59.988352 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:53:15.898584 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:53:15.898548 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:53:32.888179 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:53:32.888145 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:53:49.991145 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:53:49.991109 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:53:54.880146 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:53:54.880120 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:53:54.884293 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:53:54.884273 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:54:06.791132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:54:06.791091 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:54:22.786722 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:54:22.786684 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:54:40.593194 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:54:40.593161 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:54:55.993003 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:54:55.992970 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:55:13.593799 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:55:13.593761 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:55:33.397218 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:55:33.397138 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:55:52.805330 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:55:52.805295 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:56:11.196123 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:56:11.196092 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:56:22.199554 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:56:22.199518 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:56:43.013620 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:56:43.013587 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:57:02.488458 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:02.488421 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:57:24.896565 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:24.896531 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:57:31.346619 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.346586 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7720-140380355468848"] Jul 14 22:57:31.347087 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.346962 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" Jul 14 22:57:31.347087 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.346972 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4ac872-d541-41f2-8a01-da6dd93e9969" containerName="istio-proxy" Jul 14 22:57:31.350273 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.350235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355468848" Jul 14 22:57:31.353572 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.353549 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-784kt\"" Jul 14 22:57:31.355935 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.355911 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355468848"] Jul 14 22:57:31.452829 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.452799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/a1d87427-2266-4b3d-b894-9b26adcc65a8-kube-api-access-9dtn4\") pod \"test-curl-7720-140380355468848\" (UID: \"a1d87427-2266-4b3d-b894-9b26adcc65a8\") " pod="opendatahub/test-curl-7720-140380355468848" Jul 14 22:57:31.554133 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.554098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/a1d87427-2266-4b3d-b894-9b26adcc65a8-kube-api-access-9dtn4\") pod \"test-curl-7720-140380355468848\" (UID: \"a1d87427-2266-4b3d-b894-9b26adcc65a8\") " pod="opendatahub/test-curl-7720-140380355468848" Jul 14 22:57:31.564638 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.564616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/a1d87427-2266-4b3d-b894-9b26adcc65a8-kube-api-access-9dtn4\") pod \"test-curl-7720-140380355468848\" (UID: \"a1d87427-2266-4b3d-b894-9b26adcc65a8\") " pod="opendatahub/test-curl-7720-140380355468848" Jul 14 22:57:31.660034 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.659954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355468848" Jul 14 22:57:31.796994 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.796971 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355468848"] Jul 14 22:57:31.798822 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:57:31.798792 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1d87427_2266_4b3d_b894_9b26adcc65a8.slice/crio-f47903cc52c1e6a30359447d8a1c60280589cc1c8e7a919ac98140c690899bd7 WatchSource:0}: Error finding container f47903cc52c1e6a30359447d8a1c60280589cc1c8e7a919ac98140c690899bd7: Status 404 returned error can't find the container with id f47903cc52c1e6a30359447d8a1c60280589cc1c8e7a919ac98140c690899bd7 Jul 14 22:57:31.800135 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:31.800118 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:57:32.595358 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:32.595328 2578 generic.go:356] "Generic (PLEG): container finished" podID="a1d87427-2266-4b3d-b894-9b26adcc65a8" containerID="d9751d17f704243ee8fa31708b775dd5c8d462397dcb0d76db990012c5bdfa50" exitCode=0 Jul 14 22:57:32.595670 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:32.595404 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355468848" event={"ID":"a1d87427-2266-4b3d-b894-9b26adcc65a8","Type":"ContainerDied","Data":"d9751d17f704243ee8fa31708b775dd5c8d462397dcb0d76db990012c5bdfa50"} Jul 14 22:57:32.595670 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:32.595452 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355468848" event={"ID":"a1d87427-2266-4b3d-b894-9b26adcc65a8","Type":"ContainerStarted","Data":"f47903cc52c1e6a30359447d8a1c60280589cc1c8e7a919ac98140c690899bd7"} Jul 14 22:57:33.745881 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.745843 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355468848" Jul 14 22:57:33.785615 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.785590 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7720-140380355468848_a1d87427-2266-4b3d-b894-9b26adcc65a8/test-curl-7720-140380355468848/0.log" Jul 14 22:57:33.813594 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.813568 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7720-140380355468848"] Jul 14 22:57:33.820217 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.820196 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7720-140380355468848"] Jul 14 22:57:33.875710 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.875639 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/a1d87427-2266-4b3d-b894-9b26adcc65a8-kube-api-access-9dtn4\") pod \"a1d87427-2266-4b3d-b894-9b26adcc65a8\" (UID: \"a1d87427-2266-4b3d-b894-9b26adcc65a8\") " Jul 14 22:57:33.877849 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.877827 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d87427-2266-4b3d-b894-9b26adcc65a8-kube-api-access-9dtn4" (OuterVolumeSpecName: "kube-api-access-9dtn4") pod "a1d87427-2266-4b3d-b894-9b26adcc65a8" (UID: "a1d87427-2266-4b3d-b894-9b26adcc65a8"). InnerVolumeSpecName "kube-api-access-9dtn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:57:33.955025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.954996 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7720-140380355466672"] Jul 14 22:57:33.955449 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.955434 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d87427-2266-4b3d-b894-9b26adcc65a8" containerName="test-curl-7720-140380355468848" Jul 14 22:57:33.955501 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.955450 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d87427-2266-4b3d-b894-9b26adcc65a8" containerName="test-curl-7720-140380355468848" Jul 14 22:57:33.955541 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.955521 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d87427-2266-4b3d-b894-9b26adcc65a8" containerName="test-curl-7720-140380355468848" Jul 14 22:57:33.958549 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.958534 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355466672" Jul 14 22:57:33.967820 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.967796 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355466672"] Jul 14 22:57:33.976547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:33.976525 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dtn4\" (UniqueName: \"kubernetes.io/projected/a1d87427-2266-4b3d-b894-9b26adcc65a8-kube-api-access-9dtn4\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:57:34.077847 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:34.077815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5j56\" (UniqueName: \"kubernetes.io/projected/b1112874-27e7-4fb8-88c8-f7336d01938c-kube-api-access-r5j56\") pod \"test-curl-7720-140380355466672\" (UID: \"b1112874-27e7-4fb8-88c8-f7336d01938c\") " pod="opendatahub/test-curl-7720-140380355466672" Jul 14 22:57:34.179356 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:34.179266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5j56\" (UniqueName: \"kubernetes.io/projected/b1112874-27e7-4fb8-88c8-f7336d01938c-kube-api-access-r5j56\") pod \"test-curl-7720-140380355466672\" (UID: \"b1112874-27e7-4fb8-88c8-f7336d01938c\") " pod="opendatahub/test-curl-7720-140380355466672" Jul 14 22:57:34.189775 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:34.189749 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5j56\" (UniqueName: \"kubernetes.io/projected/b1112874-27e7-4fb8-88c8-f7336d01938c-kube-api-access-r5j56\") pod \"test-curl-7720-140380355466672\" (UID: \"b1112874-27e7-4fb8-88c8-f7336d01938c\") " pod="opendatahub/test-curl-7720-140380355466672" Jul 14 22:57:34.268107 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:34.268072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355466672" Jul 14 22:57:34.402095 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:34.402064 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355466672"] Jul 14 22:57:34.405137 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:57:34.405099 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1112874_27e7_4fb8_88c8_f7336d01938c.slice/crio-d9f191f26dc91e693e425be89f1da7c46ffe4fae8b442dcf16155c8a4d54655b WatchSource:0}: Error finding container d9f191f26dc91e693e425be89f1da7c46ffe4fae8b442dcf16155c8a4d54655b: Status 404 returned error can't find the container with id d9f191f26dc91e693e425be89f1da7c46ffe4fae8b442dcf16155c8a4d54655b Jul 14 22:57:34.607311 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:34.607269 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355466672" event={"ID":"b1112874-27e7-4fb8-88c8-f7336d01938c","Type":"ContainerStarted","Data":"d9f191f26dc91e693e425be89f1da7c46ffe4fae8b442dcf16155c8a4d54655b"} Jul 14 22:57:34.608379 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:34.608352 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355468848" Jul 14 22:57:34.608379 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:34.608374 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f47903cc52c1e6a30359447d8a1c60280589cc1c8e7a919ac98140c690899bd7" Jul 14 22:57:34.619304 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:57:34.619275 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7720-140380355468848\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="a1d87427-2266-4b3d-b894-9b26adcc65a8" pod="opendatahub/test-curl-7720-140380355468848" Jul 14 22:57:34.838421 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:57:34.838390 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7720-140380355468848\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="a1d87427-2266-4b3d-b894-9b26adcc65a8" pod="opendatahub/test-curl-7720-140380355468848" Jul 14 22:57:34.840219 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:34.840196 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d87427-2266-4b3d-b894-9b26adcc65a8" path="/var/lib/kubelet/pods/a1d87427-2266-4b3d-b894-9b26adcc65a8/volumes" Jul 14 22:57:35.615154 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:35.615120 2578 generic.go:356] "Generic (PLEG): container finished" podID="b1112874-27e7-4fb8-88c8-f7336d01938c" containerID="43d1be321140434fc8c87a11996fd716dfada38e92205b584a8bbb19a5e2c4ef" exitCode=0 Jul 14 22:57:35.615337 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:35.615192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355466672" event={"ID":"b1112874-27e7-4fb8-88c8-f7336d01938c","Type":"ContainerDied","Data":"43d1be321140434fc8c87a11996fd716dfada38e92205b584a8bbb19a5e2c4ef"} Jul 14 22:57:35.991226 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:35.991133 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:57:36.759791 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.759768 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355466672" Jul 14 22:57:36.786092 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.786064 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7720-140380355466672_b1112874-27e7-4fb8-88c8-f7336d01938c/test-curl-7720-140380355466672/0.log" Jul 14 22:57:36.820786 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.820757 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7720-140380355466672"] Jul 14 22:57:36.826525 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.826497 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7720-140380355466672"] Jul 14 22:57:36.907603 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.907520 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5j56\" (UniqueName: \"kubernetes.io/projected/b1112874-27e7-4fb8-88c8-f7336d01938c-kube-api-access-r5j56\") pod \"b1112874-27e7-4fb8-88c8-f7336d01938c\" (UID: \"b1112874-27e7-4fb8-88c8-f7336d01938c\") " Jul 14 22:57:36.909668 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.909631 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1112874-27e7-4fb8-88c8-f7336d01938c-kube-api-access-r5j56" (OuterVolumeSpecName: "kube-api-access-r5j56") pod "b1112874-27e7-4fb8-88c8-f7336d01938c" (UID: "b1112874-27e7-4fb8-88c8-f7336d01938c"). InnerVolumeSpecName "kube-api-access-r5j56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:57:36.994299 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.994265 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:36.994682 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.994668 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1112874-27e7-4fb8-88c8-f7336d01938c" containerName="test-curl-7720-140380355466672" Jul 14 22:57:36.994729 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.994683 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1112874-27e7-4fb8-88c8-f7336d01938c" containerName="test-curl-7720-140380355466672" Jul 14 22:57:36.994765 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.994755 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1112874-27e7-4fb8-88c8-f7336d01938c" containerName="test-curl-7720-140380355466672" Jul 14 22:57:36.997734 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:36.997718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:37.007273 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.007147 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:37.010270 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.010228 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5j56\" (UniqueName: \"kubernetes.io/projected/b1112874-27e7-4fb8-88c8-f7336d01938c-kube-api-access-r5j56\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:57:37.111651 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.111613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n46j\" (UniqueName: \"kubernetes.io/projected/608a06f1-82e0-4d5a-bc73-2e2a705c769f-kube-api-access-9n46j\") pod \"test-curl-7720-140380355504720\" (UID: \"608a06f1-82e0-4d5a-bc73-2e2a705c769f\") " pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:37.212408 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.212322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n46j\" (UniqueName: \"kubernetes.io/projected/608a06f1-82e0-4d5a-bc73-2e2a705c769f-kube-api-access-9n46j\") pod \"test-curl-7720-140380355504720\" (UID: \"608a06f1-82e0-4d5a-bc73-2e2a705c769f\") " pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:37.222132 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.222106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n46j\" (UniqueName: \"kubernetes.io/projected/608a06f1-82e0-4d5a-bc73-2e2a705c769f-kube-api-access-9n46j\") pod \"test-curl-7720-140380355504720\" (UID: \"608a06f1-82e0-4d5a-bc73-2e2a705c769f\") " pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:37.310592 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.310563 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:37.444393 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.444364 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:37.445741 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:57:37.445716 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod608a06f1_82e0_4d5a_bc73_2e2a705c769f.slice/crio-a55b81ffbade50a0e526d4462ea96ed6b94266b7a1fb73e3e38f6064ef494448 WatchSource:0}: Error finding container a55b81ffbade50a0e526d4462ea96ed6b94266b7a1fb73e3e38f6064ef494448: Status 404 returned error can't find the container with id a55b81ffbade50a0e526d4462ea96ed6b94266b7a1fb73e3e38f6064ef494448 Jul 14 22:57:37.627128 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.627098 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355466672" Jul 14 22:57:37.627128 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.627118 2578 scope.go:117] "RemoveContainer" containerID="43d1be321140434fc8c87a11996fd716dfada38e92205b584a8bbb19a5e2c4ef" Jul 14 22:57:37.628553 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:37.628523 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355504720" event={"ID":"608a06f1-82e0-4d5a-bc73-2e2a705c769f","Type":"ContainerStarted","Data":"a55b81ffbade50a0e526d4462ea96ed6b94266b7a1fb73e3e38f6064ef494448"} Jul 14 22:57:38.635436 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:38.635395 2578 generic.go:356] "Generic (PLEG): container finished" podID="608a06f1-82e0-4d5a-bc73-2e2a705c769f" containerID="759d51cfc413effd8566414f198df1dc3a984a4ce006d096d49ef67778cd2807" exitCode=0 Jul 14 22:57:38.635871 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:38.635478 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355504720" event={"ID":"608a06f1-82e0-4d5a-bc73-2e2a705c769f","Type":"ContainerDied","Data":"759d51cfc413effd8566414f198df1dc3a984a4ce006d096d49ef67778cd2807"} Jul 14 22:57:38.838854 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:38.838821 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1112874-27e7-4fb8-88c8-f7336d01938c" path="/var/lib/kubelet/pods/b1112874-27e7-4fb8-88c8-f7336d01938c/volumes" Jul 14 22:57:39.775706 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.775678 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:39.799614 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.799584 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7720-140380355504720_608a06f1-82e0-4d5a-bc73-2e2a705c769f/test-curl-7720-140380355504720/0.log" Jul 14 22:57:39.824772 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.824746 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:39.826885 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.826865 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:39.938458 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.938380 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n46j\" (UniqueName: \"kubernetes.io/projected/608a06f1-82e0-4d5a-bc73-2e2a705c769f-kube-api-access-9n46j\") pod \"608a06f1-82e0-4d5a-bc73-2e2a705c769f\" (UID: \"608a06f1-82e0-4d5a-bc73-2e2a705c769f\") " Jul 14 22:57:39.940402 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.940374 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608a06f1-82e0-4d5a-bc73-2e2a705c769f-kube-api-access-9n46j" (OuterVolumeSpecName: "kube-api-access-9n46j") pod "608a06f1-82e0-4d5a-bc73-2e2a705c769f" (UID: "608a06f1-82e0-4d5a-bc73-2e2a705c769f"). InnerVolumeSpecName "kube-api-access-9n46j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:57:39.957300 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.957278 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:39.957718 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.957701 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="608a06f1-82e0-4d5a-bc73-2e2a705c769f" containerName="test-curl-7720-140380355504720" Jul 14 22:57:39.957789 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.957720 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="608a06f1-82e0-4d5a-bc73-2e2a705c769f" containerName="test-curl-7720-140380355504720" Jul 14 22:57:39.957846 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.957836 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="608a06f1-82e0-4d5a-bc73-2e2a705c769f" containerName="test-curl-7720-140380355504720" Jul 14 22:57:39.961027 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.961006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:39.966670 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:39.966648 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:40.039626 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.039591 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f8rr\" (UniqueName: \"kubernetes.io/projected/7b886c84-e2a8-4bc8-be5e-ff37660ec111-kube-api-access-5f8rr\") pod \"test-curl-7720-140380355504720\" (UID: \"7b886c84-e2a8-4bc8-be5e-ff37660ec111\") " pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:40.039789 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.039769 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n46j\" (UniqueName: \"kubernetes.io/projected/608a06f1-82e0-4d5a-bc73-2e2a705c769f-kube-api-access-9n46j\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:57:40.140919 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.140882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f8rr\" (UniqueName: \"kubernetes.io/projected/7b886c84-e2a8-4bc8-be5e-ff37660ec111-kube-api-access-5f8rr\") pod \"test-curl-7720-140380355504720\" (UID: \"7b886c84-e2a8-4bc8-be5e-ff37660ec111\") " pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:40.149735 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.149706 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f8rr\" (UniqueName: \"kubernetes.io/projected/7b886c84-e2a8-4bc8-be5e-ff37660ec111-kube-api-access-5f8rr\") pod \"test-curl-7720-140380355504720\" (UID: \"7b886c84-e2a8-4bc8-be5e-ff37660ec111\") " pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:40.270709 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.270631 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:40.421053 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.421021 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:40.424324 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:57:40.424295 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b886c84_e2a8_4bc8_be5e_ff37660ec111.slice/crio-2a6cf338f28e773b94c4686deaf5e8997b3b5f90cca7f52d245ca93f94f92f2a WatchSource:0}: Error finding container 2a6cf338f28e773b94c4686deaf5e8997b3b5f90cca7f52d245ca93f94f92f2a: Status 404 returned error can't find the container with id 2a6cf338f28e773b94c4686deaf5e8997b3b5f90cca7f52d245ca93f94f92f2a Jul 14 22:57:40.648153 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.648114 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355504720" event={"ID":"7b886c84-e2a8-4bc8-be5e-ff37660ec111","Type":"ContainerStarted","Data":"2a6cf338f28e773b94c4686deaf5e8997b3b5f90cca7f52d245ca93f94f92f2a"} Jul 14 22:57:40.649296 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.649279 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:40.649354 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.649303 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a55b81ffbade50a0e526d4462ea96ed6b94266b7a1fb73e3e38f6064ef494448" Jul 14 22:57:40.663978 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.663941 2578 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7720-140380355504720" oldPodUID="608a06f1-82e0-4d5a-bc73-2e2a705c769f" podUID="7b886c84-e2a8-4bc8-be5e-ff37660ec111" Jul 14 22:57:40.840844 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:40.840809 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="608a06f1-82e0-4d5a-bc73-2e2a705c769f" path="/var/lib/kubelet/pods/608a06f1-82e0-4d5a-bc73-2e2a705c769f/volumes" Jul 14 22:57:41.656647 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:41.656609 2578 generic.go:356] "Generic (PLEG): container finished" podID="7b886c84-e2a8-4bc8-be5e-ff37660ec111" containerID="3ac93269bc66b5ee1c57a92b08e7d4ae22beaf13fd5f87f1265a928bd8a1d1a0" exitCode=0 Jul 14 22:57:41.656829 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:41.656700 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355504720" event={"ID":"7b886c84-e2a8-4bc8-be5e-ff37660ec111","Type":"ContainerDied","Data":"3ac93269bc66b5ee1c57a92b08e7d4ae22beaf13fd5f87f1265a928bd8a1d1a0"} Jul 14 22:57:42.802186 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.802164 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:42.824307 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.824264 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7720-140380355504720_7b886c84-e2a8-4bc8-be5e-ff37660ec111/test-curl-7720-140380355504720/0.log" Jul 14 22:57:42.851115 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.851090 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:42.856905 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.856883 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7720-140380355504720"] Jul 14 22:57:42.967074 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.967001 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f8rr\" (UniqueName: \"kubernetes.io/projected/7b886c84-e2a8-4bc8-be5e-ff37660ec111-kube-api-access-5f8rr\") pod \"7b886c84-e2a8-4bc8-be5e-ff37660ec111\" (UID: \"7b886c84-e2a8-4bc8-be5e-ff37660ec111\") " Jul 14 22:57:42.968994 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.968968 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b886c84-e2a8-4bc8-be5e-ff37660ec111-kube-api-access-5f8rr" (OuterVolumeSpecName: "kube-api-access-5f8rr") pod "7b886c84-e2a8-4bc8-be5e-ff37660ec111" (UID: "7b886c84-e2a8-4bc8-be5e-ff37660ec111"). InnerVolumeSpecName "kube-api-access-5f8rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:57:42.991830 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.991806 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7720-140380355505728"] Jul 14 22:57:42.992181 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.992169 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b886c84-e2a8-4bc8-be5e-ff37660ec111" containerName="test-curl-7720-140380355504720" Jul 14 22:57:42.992229 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.992183 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b886c84-e2a8-4bc8-be5e-ff37660ec111" containerName="test-curl-7720-140380355504720" Jul 14 22:57:42.992293 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.992283 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b886c84-e2a8-4bc8-be5e-ff37660ec111" containerName="test-curl-7720-140380355504720" Jul 14 22:57:42.995362 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:42.995346 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355505728" Jul 14 22:57:43.002349 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.002330 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355505728"] Jul 14 22:57:43.067839 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.067809 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f8rr\" (UniqueName: \"kubernetes.io/projected/7b886c84-e2a8-4bc8-be5e-ff37660ec111-kube-api-access-5f8rr\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:57:43.169312 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.169281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jqm\" (UniqueName: \"kubernetes.io/projected/922d3531-1183-4840-a6dc-2d5bd8ee1eda-kube-api-access-x9jqm\") pod \"test-curl-7720-140380355505728\" (UID: \"922d3531-1183-4840-a6dc-2d5bd8ee1eda\") " pod="opendatahub/test-curl-7720-140380355505728" Jul 14 22:57:43.270041 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.269964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jqm\" (UniqueName: \"kubernetes.io/projected/922d3531-1183-4840-a6dc-2d5bd8ee1eda-kube-api-access-x9jqm\") pod \"test-curl-7720-140380355505728\" (UID: \"922d3531-1183-4840-a6dc-2d5bd8ee1eda\") " pod="opendatahub/test-curl-7720-140380355505728" Jul 14 22:57:43.278652 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.278622 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jqm\" (UniqueName: \"kubernetes.io/projected/922d3531-1183-4840-a6dc-2d5bd8ee1eda-kube-api-access-x9jqm\") pod \"test-curl-7720-140380355505728\" (UID: \"922d3531-1183-4840-a6dc-2d5bd8ee1eda\") " pod="opendatahub/test-curl-7720-140380355505728" Jul 14 22:57:43.304614 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.304581 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355505728" Jul 14 22:57:43.463535 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.463440 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7720-140380355505728"] Jul 14 22:57:43.465692 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:57:43.465666 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod922d3531_1183_4840_a6dc_2d5bd8ee1eda.slice/crio-1e098fb218af02c715203047667826b312e0e0e5ac6009549b20375fb32a90ab WatchSource:0}: Error finding container 1e098fb218af02c715203047667826b312e0e0e5ac6009549b20375fb32a90ab: Status 404 returned error can't find the container with id 1e098fb218af02c715203047667826b312e0e0e5ac6009549b20375fb32a90ab Jul 14 22:57:43.674221 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.674193 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6cf338f28e773b94c4686deaf5e8997b3b5f90cca7f52d245ca93f94f92f2a" Jul 14 22:57:43.674221 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.674203 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:43.675815 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:43.675782 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355505728" event={"ID":"922d3531-1183-4840-a6dc-2d5bd8ee1eda","Type":"ContainerStarted","Data":"1e098fb218af02c715203047667826b312e0e0e5ac6009549b20375fb32a90ab"} Jul 14 22:57:43.686346 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:57:43.686317 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7720-140380355504720\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="7b886c84-e2a8-4bc8-be5e-ff37660ec111" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:44.682448 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:44.682418 2578 generic.go:356] "Generic (PLEG): container finished" podID="922d3531-1183-4840-a6dc-2d5bd8ee1eda" containerID="433a447dd9218dd85ec1f6baedae70b387d1d5d9b0bdac79a7fa367149b10f50" exitCode=0 Jul 14 22:57:44.682827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:44.682508 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7720-140380355505728" event={"ID":"922d3531-1183-4840-a6dc-2d5bd8ee1eda","Type":"ContainerDied","Data":"433a447dd9218dd85ec1f6baedae70b387d1d5d9b0bdac79a7fa367149b10f50"} Jul 14 22:57:44.703055 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:57:44.703023 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7720-140380355504720\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="7b886c84-e2a8-4bc8-be5e-ff37660ec111" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:44.839749 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:44.839709 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b886c84-e2a8-4bc8-be5e-ff37660ec111" path="/var/lib/kubelet/pods/7b886c84-e2a8-4bc8-be5e-ff37660ec111/volumes" Jul 14 22:57:44.841833 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:57:44.841804 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7720-140380355504720\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="7b886c84-e2a8-4bc8-be5e-ff37660ec111" pod="opendatahub/test-curl-7720-140380355504720" Jul 14 22:57:45.827605 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:45.827585 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355505728" Jul 14 22:57:45.858807 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:45.858776 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7720-140380355505728_922d3531-1183-4840-a6dc-2d5bd8ee1eda/test-curl-7720-140380355505728/0.log" Jul 14 22:57:45.892153 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:45.892128 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7720-140380355505728"] Jul 14 22:57:45.898592 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:45.898572 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7720-140380355505728"] Jul 14 22:57:45.995044 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:45.994966 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9jqm\" (UniqueName: \"kubernetes.io/projected/922d3531-1183-4840-a6dc-2d5bd8ee1eda-kube-api-access-x9jqm\") pod \"922d3531-1183-4840-a6dc-2d5bd8ee1eda\" (UID: \"922d3531-1183-4840-a6dc-2d5bd8ee1eda\") " Jul 14 22:57:45.997012 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:45.996982 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922d3531-1183-4840-a6dc-2d5bd8ee1eda-kube-api-access-x9jqm" (OuterVolumeSpecName: "kube-api-access-x9jqm") pod "922d3531-1183-4840-a6dc-2d5bd8ee1eda" (UID: "922d3531-1183-4840-a6dc-2d5bd8ee1eda"). InnerVolumeSpecName "kube-api-access-x9jqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:57:46.096315 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:46.096282 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9jqm\" (UniqueName: \"kubernetes.io/projected/922d3531-1183-4840-a6dc-2d5bd8ee1eda-kube-api-access-x9jqm\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:57:46.695123 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:46.695092 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7720-140380355505728" Jul 14 22:57:46.695123 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:46.695111 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e098fb218af02c715203047667826b312e0e0e5ac6009549b20375fb32a90ab" Jul 14 22:57:46.705946 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:57:46.705916 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7720-140380355505728\" is forbidden: User \"system:node:ip-10-0-131-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-45.ec2.internal' and this object" podUID="922d3531-1183-4840-a6dc-2d5bd8ee1eda" pod="opendatahub/test-curl-7720-140380355505728" Jul 14 22:57:46.837555 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:57:46.837524 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="922d3531-1183-4840-a6dc-2d5bd8ee1eda" path="/var/lib/kubelet/pods/922d3531-1183-4840-a6dc-2d5bd8ee1eda/volumes" Jul 14 22:58:02.418988 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:02.418953 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:58:15.477698 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:15.477660 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv"] Jul 14 22:58:15.478148 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:15.477854 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" containerID="cri-o://d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7" gracePeriod=30 Jul 14 22:58:20.739264 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.739226 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:58:20.815453 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815424 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-podinfo\") pod \"bdff0e37-5b13-4064-816b-ff1695b5acee\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " Jul 14 22:58:20.815619 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815472 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-token\") pod \"bdff0e37-5b13-4064-816b-ff1695b5acee\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " Jul 14 22:58:20.815619 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815502 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4njfl\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-kube-api-access-4njfl\") pod \"bdff0e37-5b13-4064-816b-ff1695b5acee\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " Jul 14 22:58:20.815619 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815527 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-socket\") pod \"bdff0e37-5b13-4064-816b-ff1695b5acee\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " Jul 14 22:58:20.815619 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815559 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-credential-socket\") pod \"bdff0e37-5b13-4064-816b-ff1695b5acee\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " Jul 14 22:58:20.815619 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815617 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-certs\") pod \"bdff0e37-5b13-4064-816b-ff1695b5acee\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " Jul 14 22:58:20.815889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815651 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-data\") pod \"bdff0e37-5b13-4064-816b-ff1695b5acee\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " Jul 14 22:58:20.815889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815684 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdff0e37-5b13-4064-816b-ff1695b5acee-istiod-ca-cert\") pod \"bdff0e37-5b13-4064-816b-ff1695b5acee\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " Jul 14 22:58:20.815889 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815722 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-envoy\") pod \"bdff0e37-5b13-4064-816b-ff1695b5acee\" (UID: \"bdff0e37-5b13-4064-816b-ff1695b5acee\") " Jul 14 22:58:20.816055 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815888 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bdff0e37-5b13-4064-816b-ff1695b5acee" (UID: "bdff0e37-5b13-4064-816b-ff1695b5acee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:20.816055 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815938 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bdff0e37-5b13-4064-816b-ff1695b5acee" (UID: "bdff0e37-5b13-4064-816b-ff1695b5acee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:20.816055 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.815946 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bdff0e37-5b13-4064-816b-ff1695b5acee" (UID: "bdff0e37-5b13-4064-816b-ff1695b5acee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:20.816216 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.816113 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:20.816216 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.816134 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:20.816216 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.816149 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:20.816406 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.816266 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdff0e37-5b13-4064-816b-ff1695b5acee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bdff0e37-5b13-4064-816b-ff1695b5acee" (UID: "bdff0e37-5b13-4064-816b-ff1695b5acee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:58:20.816748 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.816727 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-data" (OuterVolumeSpecName: "istio-data") pod "bdff0e37-5b13-4064-816b-ff1695b5acee" (UID: "bdff0e37-5b13-4064-816b-ff1695b5acee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:20.817775 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.817746 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-kube-api-access-4njfl" (OuterVolumeSpecName: "kube-api-access-4njfl") pod "bdff0e37-5b13-4064-816b-ff1695b5acee" (UID: "bdff0e37-5b13-4064-816b-ff1695b5acee"). InnerVolumeSpecName "kube-api-access-4njfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:58:20.817872 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.817778 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bdff0e37-5b13-4064-816b-ff1695b5acee" (UID: "bdff0e37-5b13-4064-816b-ff1695b5acee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:58:20.818034 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.818017 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-token" (OuterVolumeSpecName: "istio-token") pod "bdff0e37-5b13-4064-816b-ff1695b5acee" (UID: "bdff0e37-5b13-4064-816b-ff1695b5acee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:58:20.818084 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.818035 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bdff0e37-5b13-4064-816b-ff1695b5acee" (UID: "bdff0e37-5b13-4064-816b-ff1695b5acee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:20.883131 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.883101 2578 generic.go:356] "Generic (PLEG): container finished" podID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerID="d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7" exitCode=0 Jul 14 22:58:20.883233 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.883148 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" Jul 14 22:58:20.883233 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.883162 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" event={"ID":"bdff0e37-5b13-4064-816b-ff1695b5acee","Type":"ContainerDied","Data":"d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7"} Jul 14 22:58:20.883233 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.883200 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv" event={"ID":"bdff0e37-5b13-4064-816b-ff1695b5acee","Type":"ContainerDied","Data":"60ffa0233e1bc26514bea164f8d4e1339431a03aaadd3f592f43280c73d8381c"} Jul 14 22:58:20.883233 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.883222 2578 scope.go:117] "RemoveContainer" containerID="d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7" Jul 14 22:58:20.892171 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.892155 2578 scope.go:117] "RemoveContainer" containerID="06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad" Jul 14 22:58:20.900282 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.900260 2578 scope.go:117] "RemoveContainer" containerID="d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7" Jul 14 22:58:20.900551 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:58:20.900531 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7\": container with ID starting with d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7 not found: ID does not exist" containerID="d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7" Jul 14 22:58:20.900629 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.900565 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7"} err="failed to get container status \"d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7\": rpc error: code = NotFound desc = could not find container \"d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7\": container with ID starting with d036fda4c8f2382821830b5f79de2f6f852cf46a8ada27e0e62eed67951311c7 not found: ID does not exist" Jul 14 22:58:20.900629 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.900589 2578 scope.go:117] "RemoveContainer" containerID="06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad" Jul 14 22:58:20.900836 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:58:20.900817 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad\": container with ID starting with 06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad not found: ID does not exist" containerID="06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad" Jul 14 22:58:20.900890 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.900843 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad"} err="failed to get container status \"06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad\": rpc error: code = NotFound desc = could not find container \"06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad\": container with ID starting with 06129d229f235ab5bcd576d38e830c575409c39c2d278024009c3ce549c024ad not found: ID does not exist" Jul 14 22:58:20.901724 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.901706 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv"] Jul 14 22:58:20.905730 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.905712 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-7ab9a6-openshift-default-79b66bc5c5-5r8kv"] Jul 14 22:58:20.916700 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.916681 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:20.916775 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.916702 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njfl\" (UniqueName: \"kubernetes.io/projected/bdff0e37-5b13-4064-816b-ff1695b5acee-kube-api-access-4njfl\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:20.916775 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.916711 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:20.916775 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.916720 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdff0e37-5b13-4064-816b-ff1695b5acee-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:20.916775 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.916727 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:20.916775 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:20.916735 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdff0e37-5b13-4064-816b-ff1695b5acee-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:22.837286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:22.837255 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" path="/var/lib/kubelet/pods/bdff0e37-5b13-4064-816b-ff1695b5acee/volumes" Jul 14 22:58:29.788963 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:29.788914 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nxvr2"] Jul 14 22:58:41.693398 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:41.693360 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65"] Jul 14 22:58:41.693960 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:41.693643 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" containerID="cri-o://c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd" gracePeriod=30 Jul 14 22:58:42.430179 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:42.430148 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b854c6cf6-xldwt_01d52704-bc50-4e60-acd2-b40efc2b3bd5/authorino/0.log" Jul 14 22:58:44.111868 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:44.111832 2578 patch_prober.go:28] interesting pod/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:58:44.112235 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:44.111909 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:58:46.955405 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:46.955352 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:58:47.024475 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.024442 2578 generic.go:356] "Generic (PLEG): container finished" podID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerID="c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd" exitCode=0 Jul 14 22:58:47.024824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.024484 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" Jul 14 22:58:47.024824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.024517 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" event={"ID":"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6","Type":"ContainerDied","Data":"c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd"} Jul 14 22:58:47.024824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.024566 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65" event={"ID":"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6","Type":"ContainerDied","Data":"55a1e1a27f90acc92d9c331145fe9df3eb63909c7a03119801664933f0d8fbbc"} Jul 14 22:58:47.024824 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.024588 2578 scope.go:117] "RemoveContainer" containerID="c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd" Jul 14 22:58:47.033111 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033088 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-socket\") pod \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " Jul 14 22:58:47.033304 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033130 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-token\") pod \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " Jul 14 22:58:47.033304 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033159 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-envoy\") pod \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " Jul 14 22:58:47.033304 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033200 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istiod-ca-cert\") pod \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " Jul 14 22:58:47.033304 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033228 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-certs\") pod \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " Jul 14 22:58:47.033532 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033388 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" (UID: "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:47.033532 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033477 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-credential-socket\") pod \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " Jul 14 22:58:47.033532 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033512 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5zrl\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-kube-api-access-h5zrl\") pod \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " Jul 14 22:58:47.033688 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033584 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-data\") pod \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " Jul 14 22:58:47.033688 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033648 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-podinfo\") pod \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\" (UID: \"a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6\") " Jul 14 22:58:47.033810 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033596 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" (UID: "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:47.033810 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033685 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" (UID: "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:47.033810 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033721 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" (UID: "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:58:47.034025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.033929 2578 scope.go:117] "RemoveContainer" containerID="95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9" Jul 14 22:58:47.034025 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.034016 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istiod-ca-cert\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:47.034135 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.034034 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-certs\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:47.034135 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.034047 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-credential-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:47.034135 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.034060 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-workload-socket\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:47.034616 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.034592 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" (UID: "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:47.035769 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.035746 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" (UID: "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:58:47.035953 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.035928 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" (UID: "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:58:47.036036 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.036008 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-kube-api-access-h5zrl" (OuterVolumeSpecName: "kube-api-access-h5zrl") pod "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" (UID: "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6"). InnerVolumeSpecName "kube-api-access-h5zrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:58:47.036084 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.036051 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" (UID: "a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 22:58:47.055664 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.055647 2578 scope.go:117] "RemoveContainer" containerID="c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd" Jul 14 22:58:47.055948 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:58:47.055925 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd\": container with ID starting with c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd not found: ID does not exist" containerID="c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd" Jul 14 22:58:47.056018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.055959 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd"} err="failed to get container status \"c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd\": rpc error: code = NotFound desc = could not find container \"c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd\": container with ID starting with c6ceae87b943f8edc29ecc12e36f48f0c58ad308746938a9a87e921fc2c231cd not found: ID does not exist" Jul 14 22:58:47.056018 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.055982 2578 scope.go:117] "RemoveContainer" containerID="95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9" Jul 14 22:58:47.056264 ip-10-0-131-45 kubenswrapper[2578]: E0714 22:58:47.056225 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9\": container with ID starting with 95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9 not found: ID does not exist" containerID="95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9" Jul 14 22:58:47.056318 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.056263 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9"} err="failed to get container status \"95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9\": rpc error: code = NotFound desc = could not find container \"95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9\": container with ID starting with 95793fc76579a65014dbc3d19f03a0c337eb2ff75c6f5d3a7418b34c37b633d9 not found: ID does not exist" Jul 14 22:58:47.134879 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.134851 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5zrl\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-kube-api-access-h5zrl\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:47.134879 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.134877 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-data\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:47.135033 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.134893 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-podinfo\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:47.135033 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.134901 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-token\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:47.135033 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.134908 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6-istio-envoy\") on node \"ip-10-0-131-45.ec2.internal\" DevicePath \"\"" Jul 14 22:58:47.349103 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.349076 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65"] Jul 14 22:58:47.352763 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.352740 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-546296-openshift-default-6c99ddd65f-95v65"] Jul 14 22:58:47.670101 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:47.669999 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-28rcq_22f96003-9eb5-47dd-a1c4-c0a3df528f00/manager/0.log" Jul 14 22:58:48.062796 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:48.062772 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6pwzw_c0faf2cf-314f-4929-9641-8e500964305a/server/0.log" Jul 14 22:58:48.598196 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:48.598167 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c5998494-wqgcl_124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8/copy-manifests/0.log" Jul 14 22:58:48.606577 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:48.606553 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c5998494-wqgcl_124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8/manager/0.log" Jul 14 22:58:48.837846 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:48.837806 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" path="/var/lib/kubelet/pods/a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6/volumes" Jul 14 22:58:49.127302 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:49.127266 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734470-xmfps_675a968d-e550-46c8-aa15-f5a4b3d9eb8e/cleanup/0.log" Jul 14 22:58:49.248321 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:49.248289 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734485-6lb7g_90643614-5547-4ec1-99d9-9799c66a7679/cleanup/0.log" Jul 14 22:58:49.370600 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:49.370580 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-2b6fcd6d-v9gzh_97429e91-3cf1-46db-9a32-4c7eff01843a/revoke-keys/1.log" Jul 14 22:58:49.492149 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:49.492077 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-577df277-p2bbh_e8131e82-2f07-4818-bdb3-1a14d832e6b3/revoke-keys/0.log" Jul 14 22:58:49.853587 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:49.853554 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-6205e423-b66km_7c5f4ff5-6535-4792-ba35-f30199cef036/revoke-keys/0.log" Jul 14 22:58:49.976814 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:49.976781 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-fcf58c-2zq4g_22849bec-7c52-4ec0-8160-67b8fdd67abf/revoke-keys/0.log" Jul 14 22:58:50.221084 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:50.220999 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-91e25165-9zs82_f284f1df-9800-4568-9570-8aeab9f289a7/revoke-keys/0.log" Jul 14 22:58:50.342058 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:50.342031 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-7a211570-28v7n_c838f78f-ff88-4e13-b162-ed646c8665c5/revoke-keys/0.log" Jul 14 22:58:50.472387 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:50.472319 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-de2bcd69-6v7xp_96d3a017-24c6-403f-9c72-fb35a0815c31/revoke-keys/0.log" Jul 14 22:58:52.244803 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:52.244775 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b854c6cf6-xldwt_01d52704-bc50-4e60-acd2-b40efc2b3bd5/authorino/0.log" Jul 14 22:58:52.384590 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:52.384546 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-k9p69_0e46c6ea-0de1-48bc-b4e7-89c5901c029d/manager/0.log" Jul 14 22:58:52.512137 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:52.512043 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-tbb2f_ac70abc1-baf6-4d3d-8477-abd13a315f2a/manager/0.log" Jul 14 22:58:52.754035 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:52.754007 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gzwpr_60f91db4-18c6-4850-9d1c-cd013558f674/registry-server/0.log" Jul 14 22:58:52.885120 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:52.885060 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-89dml_7f4e2a3d-e431-4da3-b171-e798a6afa8e9/manager/0.log" Jul 14 22:58:53.023360 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:53.023329 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nxvr2_2da0ff5f-9d35-485b-a678-56f73d62ef83/limitador/0.log" Jul 14 22:58:53.542881 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:53.542848 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42sb8v_7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea/istio-proxy/0.log" Jul 14 22:58:53.794595 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:53.794474 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5ddb5db5fc-nf2xj_92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3/kube-auth-proxy/0.log" Jul 14 22:58:54.038945 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:54.038902 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rjcqf_85c6620a-8dd6-45f2-9862-a34d4e4afe86/istio-proxy/1.log" Jul 14 22:58:54.412169 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:54.412143 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b494b4dd8-57vzj_b952986a-e1d3-4ea4-86f0-6411a8a41eed/router/0.log" Jul 14 22:58:54.915316 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:54.915290 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:58:54.920882 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:54.920861 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:58:55.151093 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:55.151058 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkcf9j_03cf5653-385e-48de-9af8-74b4d56f8a0a/main/0.log" Jul 14 22:58:59.521484 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.521452 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nq2f/must-gather-dsxvz"] Jul 14 22:58:59.522016 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.521997 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922d3531-1183-4840-a6dc-2d5bd8ee1eda" containerName="test-curl-7720-140380355505728" Jul 14 22:58:59.522108 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522018 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="922d3531-1183-4840-a6dc-2d5bd8ee1eda" containerName="test-curl-7720-140380355505728" Jul 14 22:58:59.522108 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522033 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" Jul 14 22:58:59.522108 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522041 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" Jul 14 22:58:59.522108 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522057 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" Jul 14 22:58:59.522108 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522066 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" Jul 14 22:58:59.522394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522183 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" Jul 14 22:58:59.522394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522196 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" Jul 14 22:58:59.522394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522207 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" Jul 14 22:58:59.522394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522214 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" Jul 14 22:58:59.522394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522222 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="922d3531-1183-4840-a6dc-2d5bd8ee1eda" containerName="test-curl-7720-140380355505728" Jul 14 22:58:59.522394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522369 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" Jul 14 22:58:59.522394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522380 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdff0e37-5b13-4064-816b-ff1695b5acee" containerName="istio-proxy" Jul 14 22:58:59.522394 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522395 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" Jul 14 22:58:59.522776 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.522403 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c78d76-cdc2-46b2-9cd1-5a79aa4257f6" containerName="istio-proxy" Jul 14 22:58:59.527190 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.527170 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nq2f/must-gather-dsxvz" Jul 14 22:58:59.529965 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.529946 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nq2f\"/\"kube-root-ca.crt\"" Jul 14 22:58:59.530980 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.530964 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nq2f\"/\"openshift-service-ca.crt\"" Jul 14 22:58:59.530980 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.530971 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9nq2f\"/\"default-dockercfg-zqhpv\"" Jul 14 22:58:59.535472 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.535449 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nq2f/must-gather-dsxvz"] Jul 14 22:58:59.643276 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.643225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8mf\" (UniqueName: \"kubernetes.io/projected/f2b48ac8-f6a3-4675-85e7-9d4b9455aea4-kube-api-access-kd8mf\") pod \"must-gather-dsxvz\" (UID: \"f2b48ac8-f6a3-4675-85e7-9d4b9455aea4\") " pod="openshift-must-gather-9nq2f/must-gather-dsxvz" Jul 14 22:58:59.643450 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.643308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f2b48ac8-f6a3-4675-85e7-9d4b9455aea4-must-gather-output\") pod \"must-gather-dsxvz\" (UID: \"f2b48ac8-f6a3-4675-85e7-9d4b9455aea4\") " pod="openshift-must-gather-9nq2f/must-gather-dsxvz" Jul 14 22:58:59.743999 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.743956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8mf\" (UniqueName: \"kubernetes.io/projected/f2b48ac8-f6a3-4675-85e7-9d4b9455aea4-kube-api-access-kd8mf\") pod \"must-gather-dsxvz\" (UID: \"f2b48ac8-f6a3-4675-85e7-9d4b9455aea4\") " pod="openshift-must-gather-9nq2f/must-gather-dsxvz" Jul 14 22:58:59.743999 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.744009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f2b48ac8-f6a3-4675-85e7-9d4b9455aea4-must-gather-output\") pod \"must-gather-dsxvz\" (UID: \"f2b48ac8-f6a3-4675-85e7-9d4b9455aea4\") " pod="openshift-must-gather-9nq2f/must-gather-dsxvz" Jul 14 22:58:59.744364 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.744346 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f2b48ac8-f6a3-4675-85e7-9d4b9455aea4-must-gather-output\") pod \"must-gather-dsxvz\" (UID: \"f2b48ac8-f6a3-4675-85e7-9d4b9455aea4\") " pod="openshift-must-gather-9nq2f/must-gather-dsxvz" Jul 14 22:58:59.753135 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.753118 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8mf\" (UniqueName: \"kubernetes.io/projected/f2b48ac8-f6a3-4675-85e7-9d4b9455aea4-kube-api-access-kd8mf\") pod \"must-gather-dsxvz\" (UID: \"f2b48ac8-f6a3-4675-85e7-9d4b9455aea4\") " pod="openshift-must-gather-9nq2f/must-gather-dsxvz" Jul 14 22:58:59.836829 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.836808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nq2f/must-gather-dsxvz" Jul 14 22:58:59.971362 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:58:59.971337 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nq2f/must-gather-dsxvz"] Jul 14 22:58:59.973827 ip-10-0-131-45 kubenswrapper[2578]: W0714 22:58:59.973802 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2b48ac8_f6a3_4675_85e7_9d4b9455aea4.slice/crio-eb401b1b7ce74b16b824bcb6ce80d9a324d48624975c123c6bdf068e012f7c67 WatchSource:0}: Error finding container eb401b1b7ce74b16b824bcb6ce80d9a324d48624975c123c6bdf068e012f7c67: Status 404 returned error can't find the container with id eb401b1b7ce74b16b824bcb6ce80d9a324d48624975c123c6bdf068e012f7c67 Jul 14 22:59:00.094926 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:00.094828 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nq2f/must-gather-dsxvz" event={"ID":"f2b48ac8-f6a3-4675-85e7-9d4b9455aea4","Type":"ContainerStarted","Data":"eb401b1b7ce74b16b824bcb6ce80d9a324d48624975c123c6bdf068e012f7c67"} Jul 14 22:59:01.103129 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:01.103094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nq2f/must-gather-dsxvz" event={"ID":"f2b48ac8-f6a3-4675-85e7-9d4b9455aea4","Type":"ContainerStarted","Data":"ecd2bbe727397141ab36ee473bbe95aac5b330d895bcfdc45e8dfc10ed9c3cf2"} Jul 14 22:59:01.103129 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:01.103135 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nq2f/must-gather-dsxvz" event={"ID":"f2b48ac8-f6a3-4675-85e7-9d4b9455aea4","Type":"ContainerStarted","Data":"0e582fb6e9469d6f75fdd3b3684e5cd45834337d6820acb330b69038488e2f1b"} Jul 14 22:59:01.121780 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:01.121714 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nq2f/must-gather-dsxvz" podStartSLOduration=1.444583685 podStartE2EDuration="2.121693807s" totalImagesPullingTime="677.110122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:59:01.118902994 +0000 UTC m=+3006.931534635" watchObservedRunningTime="2026-07-14 22:59:01.121693807 +0000 UTC m=+3006.934325448" Jul 14 22:59:02.278285 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:02.278208 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f2j5k_514eec71-0eb2-4101-be06-0abbad141842/global-pull-secret-syncer/0.log" Jul 14 22:59:02.463706 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:02.463667 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v8ln8_ce9fa062-a9b3-4a71-a5c4-f76f22f36308/konnectivity-agent/0.log" Jul 14 22:59:02.488819 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:02.488793 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-45.ec2.internal_e71ccf9865e4302d818dc5edb3e82267/haproxy/0.log" Jul 14 22:59:07.225540 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:07.225479 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b854c6cf6-xldwt_01d52704-bc50-4e60-acd2-b40efc2b3bd5/authorino/0.log" Jul 14 22:59:07.258844 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:07.258816 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-k9p69_0e46c6ea-0de1-48bc-b4e7-89c5901c029d/manager/0.log" Jul 14 22:59:07.300675 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:07.300648 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-tbb2f_ac70abc1-baf6-4d3d-8477-abd13a315f2a/manager/0.log" Jul 14 22:59:07.384062 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:07.384022 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gzwpr_60f91db4-18c6-4850-9d1c-cd013558f674/registry-server/0.log" Jul 14 22:59:07.551992 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:07.551905 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-89dml_7f4e2a3d-e431-4da3-b171-e798a6afa8e9/manager/0.log" Jul 14 22:59:07.579900 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:07.579875 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nxvr2_2da0ff5f-9d35-485b-a678-56f73d62ef83/limitador/0.log" Jul 14 22:59:09.446083 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:09.446053 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-w2jcz_92e2a1a7-c30d-4973-a60a-a21ac171201a/cluster-monitoring-operator/0.log" Jul 14 22:59:09.649649 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:09.649610 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5btft_92c063a1-8397-4b68-9a9f-e985e7df7f7a/node-exporter/0.log" Jul 14 22:59:09.681424 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:09.681399 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5btft_92c063a1-8397-4b68-9a9f-e985e7df7f7a/kube-rbac-proxy/0.log" Jul 14 22:59:09.708339 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:09.708257 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5btft_92c063a1-8397-4b68-9a9f-e985e7df7f7a/init-textfile/0.log" Jul 14 22:59:11.824383 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:11.824350 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-jl2d5_5f7a705d-108e-43b9-8c21-1bcf8789c548/networking-console-plugin/0.log" Jul 14 22:59:12.473932 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.473898 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/2.log" Jul 14 22:59:12.479923 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.479896 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-fmwwd_dc87cee2-2ef1-4d66-977f-47f83f7cc504/console-operator/3.log" Jul 14 22:59:12.720286 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.714073 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6"] Jul 14 22:59:12.729595 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.729505 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6"] Jul 14 22:59:12.729758 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.729625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.795153 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.795112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7rc\" (UniqueName: \"kubernetes.io/projected/72256abf-2525-46fc-9ff6-a1d41a9cff3d-kube-api-access-pq7rc\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.795491 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.795468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-sys\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.795703 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.795684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-proc\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.795852 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.795831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-lib-modules\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.795970 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.795955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-podres\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.897491 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.897456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-sys\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.897925 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.897513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-proc\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.897925 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.897539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-lib-modules\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.897925 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.897559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-podres\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.897925 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.897611 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-sys\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.897925 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.897626 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-proc\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.897925 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.897616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7rc\" (UniqueName: \"kubernetes.io/projected/72256abf-2525-46fc-9ff6-a1d41a9cff3d-kube-api-access-pq7rc\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.897925 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.897729 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-podres\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.897925 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.897736 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72256abf-2525-46fc-9ff6-a1d41a9cff3d-lib-modules\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:12.908827 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:12.908795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7rc\" (UniqueName: \"kubernetes.io/projected/72256abf-2525-46fc-9ff6-a1d41a9cff3d-kube-api-access-pq7rc\") pod \"perf-node-gather-daemonset-s9zg6\" (UID: \"72256abf-2525-46fc-9ff6-a1d41a9cff3d\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:13.044542 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:13.044458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:13.205003 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:13.204810 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6"] Jul 14 22:59:13.224174 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:13.224023 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" event={"ID":"72256abf-2525-46fc-9ff6-a1d41a9cff3d","Type":"ContainerStarted","Data":"d1256bd142687af362f29ad2a2be2dc19415ded7a38bbbe8783c6a143d9419c6"} Jul 14 22:59:13.758997 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:13.758905 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-ts2bt_b093ad9b-7ff0-458a-8208-6356b7a1332b/volume-data-source-validator/0.log" Jul 14 22:59:14.234205 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:14.234164 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" event={"ID":"72256abf-2525-46fc-9ff6-a1d41a9cff3d","Type":"ContainerStarted","Data":"5a0926864d44fe1a832a060134edab97d676c5bafd885264e0ba28910479e86c"} Jul 14 22:59:14.234650 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:14.234234 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:14.251038 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:14.250969 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" podStartSLOduration=2.250948888 podStartE2EDuration="2.250948888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:59:14.249933895 +0000 UTC m=+3020.062565537" watchObservedRunningTime="2026-07-14 22:59:14.250948888 +0000 UTC m=+3020.063580529" Jul 14 22:59:14.878079 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:14.878050 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-55p9j_9dc2b4f2-7551-4c24-95c1-21388f68f67b/dns/0.log" Jul 14 22:59:14.900625 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:14.900598 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-55p9j_9dc2b4f2-7551-4c24-95c1-21388f68f67b/kube-rbac-proxy/0.log" Jul 14 22:59:15.087380 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:15.087353 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klpfw_4598f53c-2bba-424d-8c59-bec7a4697d98/dns-node-resolver/0.log" Jul 14 22:59:15.712545 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:15.712515 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lrct7_05ccb78a-325c-4951-b04d-d4b614127b78/node-ca/0.log" Jul 14 22:59:16.712572 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:16.712528 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42sb8v_7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea/istio-proxy/0.log" Jul 14 22:59:16.942504 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:16.942445 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5ddb5db5fc-nf2xj_92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3/kube-auth-proxy/0.log" Jul 14 22:59:17.000543 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:17.000471 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rjcqf_85c6620a-8dd6-45f2-9862-a34d4e4afe86/istio-proxy/0.log" Jul 14 22:59:17.038565 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:17.038534 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rjcqf_85c6620a-8dd6-45f2-9862-a34d4e4afe86/istio-proxy/1.log" Jul 14 22:59:17.155322 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:17.155271 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b494b4dd8-57vzj_b952986a-e1d3-4ea4-86f0-6411a8a41eed/router/0.log" Jul 14 22:59:17.897397 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:17.897371 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5mbjm_d0784199-9bd0-4ecf-a5d5-ef31376c45a8/serve-healthcheck-canary/0.log" Jul 14 22:59:18.462547 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:18.462506 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-6z66z_c78ebcf5-0805-48e9-b946-4c759026f110/insights-operator/0.log" Jul 14 22:59:18.463404 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:18.463382 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-6z66z_c78ebcf5-0805-48e9-b946-4c759026f110/insights-operator/1.log" Jul 14 22:59:18.629375 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:18.629343 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vqkwk_2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338/kube-rbac-proxy/0.log" Jul 14 22:59:18.652656 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:18.652627 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vqkwk_2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338/exporter/0.log" Jul 14 22:59:18.676313 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:18.676286 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vqkwk_2aae1eb9-cd7a-4a3b-bad5-a342dd3c5338/extractor/0.log" Jul 14 22:59:20.250338 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:20.250311 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-s9zg6" Jul 14 22:59:20.806775 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:20.806741 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-28rcq_22f96003-9eb5-47dd-a1c4-c0a3df528f00/manager/0.log" Jul 14 22:59:21.252406 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:21.252374 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6pwzw_c0faf2cf-314f-4929-9641-8e500964305a/server/0.log" Jul 14 22:59:21.586137 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:21.586092 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c5998494-wqgcl_124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8/manager/0.log" Jul 14 22:59:21.616722 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:21.616695 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c5998494-wqgcl_124ab4a8-e642-4e7d-9cdf-e3e9ef6b79e8/copy-manifests/0.log" Jul 14 22:59:22.692306 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:22.692214 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42sb8v_7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea/istio-proxy/0.log" Jul 14 22:59:22.907572 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:22.907544 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5ddb5db5fc-nf2xj_92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3/kube-auth-proxy/0.log" Jul 14 22:59:22.944477 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:22.944370 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rjcqf_85c6620a-8dd6-45f2-9862-a34d4e4afe86/istio-proxy/0.log" Jul 14 22:59:22.979046 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:22.979016 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rjcqf_85c6620a-8dd6-45f2-9862-a34d4e4afe86/istio-proxy/1.log" Jul 14 22:59:23.056705 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:23.056669 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b494b4dd8-57vzj_b952986a-e1d3-4ea4-86f0-6411a8a41eed/router/0.log" Jul 14 22:59:24.369265 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:24.369225 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42sb8v_7ff7a9ed-6dad-4e42-9adb-4f2a04d900ea/istio-proxy/0.log" Jul 14 22:59:24.585610 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:24.585537 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5ddb5db5fc-nf2xj_92c9bdcf-d96d-4b3f-be8c-fcf5e86d70e3/kube-auth-proxy/0.log" Jul 14 22:59:24.629083 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:24.629048 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rjcqf_85c6620a-8dd6-45f2-9862-a34d4e4afe86/istio-proxy/0.log" Jul 14 22:59:24.666961 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:24.666927 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rjcqf_85c6620a-8dd6-45f2-9862-a34d4e4afe86/istio-proxy/1.log" Jul 14 22:59:24.744931 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:24.744890 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b494b4dd8-57vzj_b952986a-e1d3-4ea4-86f0-6411a8a41eed/router/0.log" Jul 14 22:59:28.093271 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:28.093232 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-v94mh_05bb1b72-634d-4d59-a98a-d7c49e292dcf/migrator/0.log" Jul 14 22:59:28.115950 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:28.115922 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-v94mh_05bb1b72-634d-4d59-a98a-d7c49e292dcf/graceful-termination/0.log" Jul 14 22:59:28.527393 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:28.527314 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-66wfq_95fe832f-c448-4835-bdd6-2c03d94edc5f/kube-storage-version-migrator-operator/1.log" Jul 14 22:59:28.528310 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:28.528288 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-66wfq_95fe832f-c448-4835-bdd6-2c03d94edc5f/kube-storage-version-migrator-operator/0.log" Jul 14 22:59:29.556992 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:29.556961 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6zkw_aee8d65f-586f-4879-a62f-69223df24c06/kube-multus-additional-cni-plugins/0.log" Jul 14 22:59:29.579344 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:29.579314 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6zkw_aee8d65f-586f-4879-a62f-69223df24c06/egress-router-binary-copy/0.log" Jul 14 22:59:29.602530 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:29.602501 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6zkw_aee8d65f-586f-4879-a62f-69223df24c06/cni-plugins/0.log" Jul 14 22:59:29.625335 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:29.625310 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6zkw_aee8d65f-586f-4879-a62f-69223df24c06/bond-cni-plugin/0.log" Jul 14 22:59:29.647341 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:29.647277 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6zkw_aee8d65f-586f-4879-a62f-69223df24c06/routeoverride-cni/0.log" Jul 14 22:59:29.670934 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:29.670910 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6zkw_aee8d65f-586f-4879-a62f-69223df24c06/whereabouts-cni-bincopy/0.log" Jul 14 22:59:29.693669 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:29.693626 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6zkw_aee8d65f-586f-4879-a62f-69223df24c06/whereabouts-cni/0.log" Jul 14 22:59:30.065955 ip-10-0-131-45 kubenswrapper[2578]: I0714 22:59:30.065932 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v9hsx_2519ea9e-2e8c-42a1-81c8-3649662d1c09/kube-multus/0.log"