Jul 29 13:59:19.220417 ip-10-0-139-155 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 13:59:19.220431 ip-10-0-139-155 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 13:59:19.220442 ip-10-0-139-155 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 13:59:19.220809 ip-10-0-139-155 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 13:59:29.462654 ip-10-0-139-155 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 13:59:29.462678 ip-10-0-139-155 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8774ab03422e4db99b89784cdf7d3764 -- Jul 29 14:01:06.662818 ip-10-0-139-155 systemd[1]: Starting Kubernetes Kubelet... Jul 29 14:01:07.178510 ip-10-0-139-155 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 14:01:07.178510 ip-10-0-139-155 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 14:01:07.178510 ip-10-0-139-155 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 14:01:07.178510 ip-10-0-139-155 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 14:01:07.178510 ip-10-0-139-155 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 14:01:07.179281 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.178593 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 14:01:07.182353 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182330 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:07.182353 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182349 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:07.182353 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182354 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:07.182353 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182359 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182363 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182367 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182371 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182375 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182379 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182382 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182386 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182389 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182397 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182403 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182407 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182411 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182414 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182418 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182423 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182426 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182430 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182434 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:07.182627 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182441 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182446 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182451 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182457 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182462 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182466 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182470 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182474 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182478 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182482 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182487 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182491 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182495 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182499 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182503 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182509 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182514 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182518 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182522 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:07.183451 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182526 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182531 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182553 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182558 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182562 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182566 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182570 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182574 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182578 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182582 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182586 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182592 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182596 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182600 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182604 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182608 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182613 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182617 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182622 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182625 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:07.184252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182629 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182634 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182638 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182642 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182646 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182650 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182654 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182659 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182667 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182672 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182676 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182681 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182685 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182692 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182698 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182703 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182708 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182712 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182717 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182722 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:07.185113 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182726 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182731 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182735 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182740 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182744 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182748 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182753 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182757 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182761 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182765 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182769 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182773 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182777 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182781 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182785 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182789 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182793 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.182798 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184062 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:07.185989 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184074 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184080 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184085 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184089 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184094 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184099 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184103 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184108 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184113 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184117 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184122 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184126 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184132 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184138 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184143 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184147 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184152 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184156 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184160 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:07.186818 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184164 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184169 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184175 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184182 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184187 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184192 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184196 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184208 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184213 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184218 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184227 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184231 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184236 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184240 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184244 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184248 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184251 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184255 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184260 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184264 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:07.187474 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184268 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184272 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184276 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184280 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184285 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184289 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184293 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184298 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184302 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184306 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184310 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184314 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184318 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184322 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184326 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184331 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184335 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184351 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184356 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184360 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:07.188078 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184364 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184368 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184374 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184378 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184383 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184387 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184391 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184395 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184399 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184403 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184408 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184418 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184424 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184428 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184433 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184437 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184441 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184445 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184449 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184453 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:07.188642 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184457 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184461 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184465 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184469 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184473 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184478 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184482 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184486 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184491 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184495 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184499 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184506 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184511 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184515 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184521 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184525 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184529 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184554 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.184558 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185198 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:07.189349 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185207 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185211 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185216 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185221 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185225 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185230 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185234 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185239 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185243 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185247 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185251 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185256 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185260 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185264 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185268 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185272 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185277 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185281 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185285 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185289 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:07.189885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185294 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185298 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185302 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185305 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185309 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185313 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185318 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185322 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185329 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185333 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185337 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185341 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185345 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185349 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185353 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185361 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185368 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185372 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185377 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:07.190358 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185381 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185385 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185389 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185393 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185397 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185402 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185406 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185410 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185414 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185418 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185423 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185428 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185432 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185437 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185441 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185445 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185449 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185454 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185458 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185463 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:07.190969 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185467 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185473 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185477 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185481 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185486 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185490 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185494 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185498 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185502 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185506 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185511 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185515 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185519 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185523 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185527 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185532 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185554 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185558 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185562 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185566 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:07.191456 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185570 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185574 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185580 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185586 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185592 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185597 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185601 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185606 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185612 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185616 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185620 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185625 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185629 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185638 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185642 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185647 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185651 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185655 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.185659 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185758 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 14:01:07.191977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185774 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185784 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185791 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185798 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185803 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185810 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185816 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185821 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185826 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185832 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185837 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185842 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185847 2560 flags.go:64] FLAG: --cgroup-root="" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185852 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185856 2560 flags.go:64] FLAG: --client-ca-file="" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185861 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185867 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185874 2560 flags.go:64] FLAG: --cluster-domain="" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185878 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185883 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185889 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185894 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185900 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 14:01:07.192494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185904 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185909 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185917 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185921 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185926 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185931 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185936 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185941 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185947 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185952 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185957 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185961 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185966 2560 flags.go:64] FLAG: --enable-server="true" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185972 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185979 2560 flags.go:64] FLAG: --event-burst="100" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185984 2560 flags.go:64] FLAG: --event-qps="50" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185988 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185993 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.185998 2560 flags.go:64] FLAG: --eviction-hard="" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186003 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186007 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186012 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186017 2560 flags.go:64] FLAG: --eviction-soft="" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186021 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186026 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 14:01:07.193115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186033 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186038 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186042 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186047 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186052 2560 flags.go:64] FLAG: --feature-gates="" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186057 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186062 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186067 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186072 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186079 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186084 2560 flags.go:64] FLAG: --help="false" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186089 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186094 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186099 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186103 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186109 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186115 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186119 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186124 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186128 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186133 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186139 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186144 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186149 2560 flags.go:64] FLAG: --kube-reserved="" Jul 29 14:01:07.193745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186153 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186158 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186163 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186168 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186172 2560 flags.go:64] FLAG: --lock-file="" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186177 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186181 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186186 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186197 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186202 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186207 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186211 2560 flags.go:64] FLAG: --logging-format="text" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186216 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186222 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186226 2560 flags.go:64] FLAG: --manifest-url="" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186230 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186237 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186244 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186250 2560 flags.go:64] FLAG: --max-pods="110" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186255 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186260 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186265 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186269 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186274 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186279 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 14:01:07.194393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186284 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186294 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186299 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186304 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186308 2560 flags.go:64] FLAG: --pod-cidr="" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186314 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186322 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186327 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186332 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186337 2560 flags.go:64] FLAG: --port="10250" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186342 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186346 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0315f4e12f19ebf85" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186351 2560 flags.go:64] FLAG: --qos-reserved="" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186356 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186361 2560 flags.go:64] FLAG: --register-node="true" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186366 2560 flags.go:64] FLAG: --register-with-taints="" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186372 2560 flags.go:64] FLAG: --registry-burst="10" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186377 2560 flags.go:64] FLAG: --registry-qps="5" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186381 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186386 2560 flags.go:64] FLAG: --reserved-memory="" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186392 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186397 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186402 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186407 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186413 2560 flags.go:64] FLAG: --runonce="false" Jul 29 14:01:07.195131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186418 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186423 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186428 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186432 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186437 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186442 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186447 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186452 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186457 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186462 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186467 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186471 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186476 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186481 2560 flags.go:64] FLAG: --system-cgroups="" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186486 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186494 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186499 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186503 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186509 2560 flags.go:64] FLAG: --tls-min-version="" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186513 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186518 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186522 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186527 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186552 2560 flags.go:64] FLAG: --v="2" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186560 2560 flags.go:64] FLAG: --version="false" Jul 29 14:01:07.195819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186566 2560 flags.go:64] FLAG: --vmodule="" Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186572 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.186577 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186766 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186772 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186776 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186784 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186789 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186793 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186798 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186802 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186806 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186811 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186814 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186819 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186822 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186827 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186831 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186835 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186839 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:07.196449 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186843 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186848 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186852 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186857 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186861 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186866 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186870 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186874 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186878 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186882 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186890 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186897 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186904 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186910 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186915 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186920 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186924 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186928 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186934 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:07.196994 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186939 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186944 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186948 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186952 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186956 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186960 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186965 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186969 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186973 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186978 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186982 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186986 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186991 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186995 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.186999 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187003 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187008 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187012 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187016 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187020 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:07.197436 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187024 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187028 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187033 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187039 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187043 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187047 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187051 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187055 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187059 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187063 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187068 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187073 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187077 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187081 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187086 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187090 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187094 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187098 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187102 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187107 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:07.197944 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187111 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187115 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187119 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187124 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187128 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187132 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187136 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187140 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187145 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187149 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187153 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187157 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187164 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187169 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187174 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187180 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187185 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187189 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187194 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:07.198491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187198 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187202 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187207 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.187211 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.187786 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.195555 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.195572 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.195597 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.195603 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195667 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195672 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195676 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195678 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195682 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:07.199229 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195684 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195687 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195690 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195693 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195697 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195699 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195702 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195710 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195713 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195718 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195721 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195724 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195727 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195730 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195733 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195735 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195738 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195740 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195743 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195745 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:07.199809 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195748 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195751 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195753 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195756 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195759 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195768 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195771 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195773 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195776 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195779 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195782 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195784 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195787 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195789 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195792 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195794 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195797 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195799 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195802 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195806 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:07.200393 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195815 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195818 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195821 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195824 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195827 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195830 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195832 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195835 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195837 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195840 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195843 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195845 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195848 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195850 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195853 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195855 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195858 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195861 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195864 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:07.200914 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195866 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195869 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195871 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195874 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195876 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195879 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195881 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195884 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195886 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195888 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195892 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195894 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195897 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195899 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195907 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195910 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195912 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195915 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195917 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195920 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:07.201373 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195923 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195925 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195927 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195930 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195932 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195935 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195937 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195940 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195943 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195945 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195948 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195950 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195953 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195956 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.195960 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:07.201895 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.195966 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196119 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196124 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196127 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196129 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196132 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196135 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196137 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196140 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196143 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196146 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196148 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196157 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196160 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196162 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196165 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196167 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196170 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196172 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196175 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 14:01:07.202271 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196177 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196180 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196182 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196184 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196187 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196189 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196192 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196194 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196197 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196199 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196202 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196204 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196206 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196209 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196212 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196214 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196217 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196219 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196221 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196224 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 14:01:07.202782 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196226 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196229 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196231 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196234 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196236 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196244 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196246 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196249 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196251 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196254 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196256 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196259 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196261 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196263 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196266 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196268 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196270 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196273 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196276 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196278 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 14:01:07.203263 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196280 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196283 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196286 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196288 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196290 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196293 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196296 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196298 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196301 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196303 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196305 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196308 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196310 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196313 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196315 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196317 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196320 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196322 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196330 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196333 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 14:01:07.203824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196335 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196337 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196340 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196342 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196345 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196347 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196350 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196353 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196355 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196359 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196362 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196365 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196369 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196373 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196376 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196378 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196381 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196383 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 14:01:07.204329 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196386 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 14:01:07.204797 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:07.196388 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 14:01:07.204797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.196393 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 14:01:07.204797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.196499 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 14:01:07.204797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.200013 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 14:01:07.204797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.201258 2560 server.go:1019] "Starting client certificate rotation" Jul 29 14:01:07.204797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.201355 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 14:01:07.204797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.201386 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 14:01:07.204797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.204330 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 14:01:07.205618 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.205569 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 14:01:07.214132 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.214109 2560 log.go:25] "Validated CRI v1 runtime API" Jul 29 14:01:07.218356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.218341 2560 log.go:25] "Validated CRI v1 image API" Jul 29 14:01:07.220687 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.220673 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 14:01:07.225138 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.225112 2560 fs.go:135] Filesystem UUIDs: map[02fc5438-d257-4dcf-88ed-c3c00c4d960e:/dev/nvme0n1p3 5bd9472b-72f2-424c-8853-1feba64514b2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 29 14:01:07.225202 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.225138 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 14:01:07.231561 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.231419 2560 manager.go:216] Machine: {Timestamp:2026-07-29 14:01:07.230943459 +0000 UTC m=+0.432444622 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099760 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd737c9f752c264017df171e293a0 SystemUUID:ec2bd737-c9f7-52c2-6401-7df171e293a0 BootID:8774ab03-422e-4db9-9b89-784cdf7d3764 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b6:78:18:25:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b6:78:18:25:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:9a:97:6f:e4:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 14:01:07.231561 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.231529 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 14:01:07.231696 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.231681 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 14:01:07.232382 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.232365 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 14:01:07.234581 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.234529 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 14:01:07.234721 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.234582 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 14:01:07.235248 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.235238 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 14:01:07.235281 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.235252 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 14:01:07.235281 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.235259 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 14:01:07.235281 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.235270 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 14:01:07.236087 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.236076 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 14:01:07.237645 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.237634 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 29 14:01:07.237755 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.237745 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 14:01:07.237795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.237777 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 29 14:01:07.237795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.237793 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 14:01:07.237870 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.237804 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 14:01:07.237870 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.237813 2560 kubelet.go:398] "Adding apiserver pod source" Jul 29 14:01:07.237870 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.237822 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 14:01:07.238195 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.238178 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 14:01:07.238229 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.238213 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 14:01:07.240370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.240355 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 14:01:07.251190 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.251023 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 14:01:07.251190 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.251176 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 14:01:07.251694 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.251679 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 14:01:07.251747 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.251711 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 14:01:07.253515 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253504 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 14:01:07.253563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253520 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 14:01:07.253563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253527 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 14:01:07.253563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253532 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 14:01:07.253563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253557 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 14:01:07.253563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253563 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 14:01:07.253698 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253570 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 14:01:07.253698 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253576 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 14:01:07.253698 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253583 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 14:01:07.253698 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253588 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 14:01:07.253698 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253596 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 14:01:07.253698 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.253601 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 14:01:07.254340 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.254331 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 14:01:07.254340 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.254340 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 14:01:07.257421 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.257407 2560 server.go:1268] "Started kubelet" Jul 29 14:01:07.257577 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.257495 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 14:01:07.257661 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.257599 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 14:01:07.257715 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.257529 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 14:01:07.258251 ip-10-0-139-155 systemd[1]: Started Kubernetes Kubelet. Jul 29 14:01:07.259506 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.259487 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 14:01:07.260358 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.260339 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 29 14:01:07.264023 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.264005 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 14:01:07.265838 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.265822 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-155.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 14:01:07.266623 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.265792 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-155.ec2.internal.18c6c782e26bdf6d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-155.ec2.internal,UID:ip-10-0-139-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-155.ec2.internal,},FirstTimestamp:2026-07-29 14:01:07.257376621 +0000 UTC m=+0.458877785,LastTimestamp:2026-07-29 14:01:07.257376621 +0000 UTC m=+0.458877785,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-155.ec2.internal,}" Jul 29 14:01:07.268106 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.268085 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 14:01:07.268178 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.268091 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 14:01:07.268227 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.268199 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:07.268677 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.268649 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 14:01:07.268677 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.268671 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 14:01:07.268807 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.268658 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 14:01:07.268860 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.268824 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 29 14:01:07.268860 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.268835 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 29 14:01:07.269428 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.269412 2560 factory.go:153] Registering CRI-O factory Jul 29 14:01:07.269428 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.269429 2560 factory.go:223] Registration of the crio container factory successfully Jul 29 14:01:07.269527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.269507 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 14:01:07.269527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.269515 2560 factory.go:55] Registering systemd factory Jul 29 14:01:07.269527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.269521 2560 factory.go:223] Registration of the systemd container factory successfully Jul 29 14:01:07.269691 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.269566 2560 factory.go:103] Registering Raw factory Jul 29 14:01:07.269691 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.269581 2560 manager.go:1196] Started watching for new ooms in manager Jul 29 14:01:07.270225 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.270051 2560 manager.go:349] Starting recovery of all containers Jul 29 14:01:07.271554 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.271517 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 14:01:07.271554 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.271520 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 14:01:07.279365 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.279347 2560 manager.go:354] Recovery completed Jul 29 14:01:07.279453 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.279424 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q4rwq" Jul 29 14:01:07.283307 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.283295 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:07.285097 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.285080 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:07.285172 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.285113 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:07.285172 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.285127 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:07.285911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.285891 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 14:01:07.285911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.285909 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 14:01:07.286048 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.285925 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 29 14:01:07.286048 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.285980 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:07.287249 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.287230 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:07.287315 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.287263 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:07.287315 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.287278 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:07.287682 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.287666 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q4rwq" Jul 29 14:01:07.287972 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.287874 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-155.ec2.internal.18c6c782e412d87a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-155.ec2.internal,UID:ip-10-0-139-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-155.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-155.ec2.internal,},FirstTimestamp:2026-07-29 14:01:07.28509657 +0000 UTC m=+0.486597733,LastTimestamp:2026-07-29 14:01:07.28509657 +0000 UTC m=+0.486597733,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-155.ec2.internal,}" Jul 29 14:01:07.290141 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.290127 2560 policy_none.go:49] "None policy: Start" Jul 29 14:01:07.290208 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.290144 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 14:01:07.290208 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.290154 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 14:01:07.291079 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.291068 2560 policy_none.go:47] "Start" Jul 29 14:01:07.322366 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.322164 2560 manager.go:337] "Starting Device Plugin manager" Jul 29 14:01:07.337273 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.322397 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 14:01:07.337273 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.322412 2560 server.go:87] "Starting device plugin registration server" Jul 29 14:01:07.337273 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.322680 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 14:01:07.337273 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.322691 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 14:01:07.337273 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.322841 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 14:01:07.337273 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.322943 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 14:01:07.337273 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.322951 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 14:01:07.337273 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.323354 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 14:01:07.337273 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.323392 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:07.372415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.372384 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 14:01:07.373677 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.373659 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 14:01:07.373777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.373706 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 14:01:07.373777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.373728 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 14:01:07.373777 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.373766 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 14:01:07.376378 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.376355 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:07.423603 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.423515 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:07.424481 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.424464 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:07.424579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.424494 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:07.424579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.424504 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:07.424579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.424527 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.434463 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.434443 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.434510 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.434468 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-155.ec2.internal\": node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:07.474348 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.474325 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal"] Jul 29 14:01:07.474431 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.474407 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:07.475341 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.475327 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:07.475410 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.475354 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:07.475410 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.475370 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:07.476630 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.476616 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:07.476751 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.476739 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.476785 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.476771 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:07.477293 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.477280 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:07.477362 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.477307 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:07.477362 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.477321 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:07.477444 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.477308 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:07.477444 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.477386 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:07.477444 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.477398 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:07.478525 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.478512 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.478583 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.478554 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 14:01:07.479220 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.479202 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientMemory" Jul 29 14:01:07.479295 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.479227 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 14:01:07.479295 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.479241 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeHasSufficientPID" Jul 29 14:01:07.486881 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.486855 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:07.514278 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.514260 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-155.ec2.internal\" not found" node="ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.518782 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.518767 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-155.ec2.internal\" not found" node="ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.570404 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.570378 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e66e6a37cab01c0d262e04ba35d56b5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal\" (UID: \"e66e6a37cab01c0d262e04ba35d56b5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.570404 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.570405 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e66e6a37cab01c0d262e04ba35d56b5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal\" (UID: \"e66e6a37cab01c0d262e04ba35d56b5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.570603 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.570423 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cca166521caabdf057abdac110b1500-config\") pod \"kube-apiserver-proxy-ip-10-0-139-155.ec2.internal\" (UID: \"5cca166521caabdf057abdac110b1500\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.587527 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.587502 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:07.670965 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.670921 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e66e6a37cab01c0d262e04ba35d56b5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal\" (UID: \"e66e6a37cab01c0d262e04ba35d56b5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.670965 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.670969 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e66e6a37cab01c0d262e04ba35d56b5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal\" (UID: \"e66e6a37cab01c0d262e04ba35d56b5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.671145 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.670995 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cca166521caabdf057abdac110b1500-config\") pod \"kube-apiserver-proxy-ip-10-0-139-155.ec2.internal\" (UID: \"5cca166521caabdf057abdac110b1500\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.671145 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.671022 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e66e6a37cab01c0d262e04ba35d56b5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal\" (UID: \"e66e6a37cab01c0d262e04ba35d56b5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.671145 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.671040 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e66e6a37cab01c0d262e04ba35d56b5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal\" (UID: \"e66e6a37cab01c0d262e04ba35d56b5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.671145 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.671040 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cca166521caabdf057abdac110b1500-config\") pod \"kube-apiserver-proxy-ip-10-0-139-155.ec2.internal\" (UID: \"5cca166521caabdf057abdac110b1500\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.688044 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.688013 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:07.788902 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.788863 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:07.818064 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.818040 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.823092 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:07.822344 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal" Jul 29 14:01:07.889222 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.889177 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:07.989746 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:07.989684 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:08.090172 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:08.090140 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:08.160865 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.160843 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:08.190346 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:08.190321 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:08.200419 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.200399 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 14:01:08.200566 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.200529 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 14:01:08.200609 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.200579 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 14:01:08.291030 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:08.290961 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-155.ec2.internal\" not found" Jul 29 14:01:08.292070 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.292037 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 13:56:07 +0000 UTC" deadline="2028-01-25 14:22:40.019978994 +0000 UTC" Jul 29 14:01:08.292120 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.292070 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13080h21m31.727912309s" Jul 29 14:01:08.307610 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.307586 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:08.368786 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.368749 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" Jul 29 14:01:08.381150 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.381127 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 14:01:08.382968 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.382946 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal" Jul 29 14:01:08.393943 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.393912 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 14:01:08.432362 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:08.432323 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cca166521caabdf057abdac110b1500.slice/crio-8ea20a2cb2e29b8e99a9997237f84baef69c78690237076692f8a29244f0c066 WatchSource:0}: Error finding container 8ea20a2cb2e29b8e99a9997237f84baef69c78690237076692f8a29244f0c066: Status 404 returned error can't find the container with id 8ea20a2cb2e29b8e99a9997237f84baef69c78690237076692f8a29244f0c066 Jul 29 14:01:08.436747 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.436733 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:01:08.455558 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:08.455523 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66e6a37cab01c0d262e04ba35d56b5e.slice/crio-902a3172096a64abc40e2253fe41f54362b09ecacd2db9d2371c825f7afd02c3 WatchSource:0}: Error finding container 902a3172096a64abc40e2253fe41f54362b09ecacd2db9d2371c825f7afd02c3: Status 404 returned error can't find the container with id 902a3172096a64abc40e2253fe41f54362b09ecacd2db9d2371c825f7afd02c3 Jul 29 14:01:08.606696 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:08.606621 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:09.239376 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.239334 2560 apiserver.go:52] "Watching apiserver" Jul 29 14:01:09.247585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.247554 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 14:01:09.248974 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.248939 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-plvns","kube-system/global-pull-secret-syncer-2vm4f","kube-system/konnectivity-agent-hpg7p","kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal","openshift-dns/node-resolver-w5jp9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal","openshift-multus/multus-additional-cni-plugins-rrvvf","openshift-multus/network-metrics-daemon-vfv5m","openshift-network-diagnostics/network-check-target-8qt8j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69","openshift-cluster-node-tuning-operator/tuned-hthjh","openshift-image-registry/node-ca-v5hwn","openshift-multus/multus-r7xf6","openshift-network-operator/iptables-alerter-cc6lv"] Jul 29 14:01:09.251704 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.251680 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.253406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.253369 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:09.253521 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.253454 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:09.254267 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.254245 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 14:01:09.254364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.254271 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w7hfg\"" Jul 29 14:01:09.254364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.254275 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 14:01:09.254364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.254330 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 14:01:09.254802 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.254779 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.256115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.256070 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.256948 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.256926 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:01:09.257052 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.257031 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 14:01:09.257126 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.257054 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-78lgq\"" Jul 29 14:01:09.257285 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.257267 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 14:01:09.257612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.257591 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.258282 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.258262 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 14:01:09.258345 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.258284 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qck8x\"" Jul 29 14:01:09.258463 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.258444 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 14:01:09.258576 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.258557 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 14:01:09.259022 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.259002 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.260066 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.260047 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 14:01:09.260309 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.260292 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 14:01:09.260394 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.260316 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 14:01:09.260647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.260388 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 14:01:09.260647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.260478 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 14:01:09.260795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.260776 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ccv6q\"" Jul 29 14:01:09.260932 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.260912 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:01:09.261009 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.260994 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bpbqm\"" Jul 29 14:01:09.261229 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.261214 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 14:01:09.261890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.261869 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.264098 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.264081 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 14:01:09.264098 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.264093 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 14:01:09.264335 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.264317 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sgq4z\"" Jul 29 14:01:09.264408 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.264357 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.265839 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.265821 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.265938 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.265884 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:09.266585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.266261 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 14:01:09.266699 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.266644 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kwvxs\"" Jul 29 14:01:09.267374 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.267355 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.268359 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.268297 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 14:01:09.268946 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.268927 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:09.269054 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.269031 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:09.269831 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.269811 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 14:01:09.269831 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.269821 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 14:01:09.270150 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.270130 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 14:01:09.270244 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.270166 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xgc9t\"" Jul 29 14:01:09.270495 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.270468 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:09.270593 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.270542 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 14:01:09.270659 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.270623 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 14:01:09.270659 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.270652 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 14:01:09.272857 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.272823 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 14:01:09.273120 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.273105 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4dnrb\"" Jul 29 14:01:09.273183 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.273130 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 14:01:09.280456 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280432 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-run-netns\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.280577 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280468 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-run-netns\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.280577 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280496 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-slash\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.280577 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280522 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-run-openvswitch\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.280719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280547 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/721cdb29-0d68-4127-b176-0f7d020a53c4-ovnkube-config\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.280719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8c01759-3d25-4362-922e-dc595a65771e-agent-certs\") pod \"konnectivity-agent-hpg7p\" (UID: \"d8c01759-3d25-4362-922e-dc595a65771e\") " pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:09.280719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280653 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-modprobe-d\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.280719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280677 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-sysctl-conf\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.280719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-systemd\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.280922 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280727 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-host\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.280922 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280752 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.280922 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280803 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.280922 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280828 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrxm\" (UniqueName: \"kubernetes.io/projected/e9a1dbd3-0eba-4141-a243-bfb4b7da0740-kube-api-access-jxrxm\") pod \"node-ca-v5hwn\" (UID: \"e9a1dbd3-0eba-4141-a243-bfb4b7da0740\") " pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.280922 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-sys\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.280922 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280874 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-lib-modules\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.280922 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280890 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19853864-4d88-4abe-8485-aa693e06ab08-tmp-dir\") pod \"node-resolver-w5jp9\" (UID: \"19853864-4d88-4abe-8485-aa693e06ab08\") " pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280929 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-run-systemd\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280954 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-var-lib-openvswitch\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.280984 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-run-ovn\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281028 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/721cdb29-0d68-4127-b176-0f7d020a53c4-env-overrides\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281068 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-sysctl-d\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281089 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281106 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-system-cni-dir\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281126 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/721cdb29-0d68-4127-b176-0f7d020a53c4-ovn-node-metrics-cert\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281156 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9a1dbd3-0eba-4141-a243-bfb4b7da0740-serviceca\") pod \"node-ca-v5hwn\" (UID: \"e9a1dbd3-0eba-4141-a243-bfb4b7da0740\") " pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.281198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281180 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ebe845f-c3d6-4eed-a05e-88b608cc827f-dbus\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281201 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34a812d1-d5c3-4506-9892-351debc35566-cni-binary-copy\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281240 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34a812d1-d5c3-4506-9892-351debc35566-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281260 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-multus-cni-dir\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281275 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-cnibin\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281300 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-multus-socket-dir-parent\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281320 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-var-lib-kubelet\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281341 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-os-release\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281359 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b97d4e52-96f4-4464-ad58-a03904d77083-cni-binary-copy\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281383 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-hostroot\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281409 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-multus-conf-dir\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281424 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-kubelet\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281438 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-var-lib-kubelet\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281456 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-tuned\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281479 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19853864-4d88-4abe-8485-aa693e06ab08-hosts-file\") pod \"node-resolver-w5jp9\" (UID: \"19853864-4d88-4abe-8485-aa693e06ab08\") " pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281509 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-socket-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281530 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.281559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281546 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-run\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281570 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctv6\" (UniqueName: \"kubernetes.io/projected/b97d4e52-96f4-4464-ad58-a03904d77083-kube-api-access-dctv6\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281606 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-cni-netd\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281621 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281641 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-sysconfig\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281663 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281685 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34a812d1-d5c3-4506-9892-351debc35566-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281701 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-etc-kubernetes\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281718 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twk52\" (UniqueName: \"kubernetes.io/projected/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-kube-api-access-twk52\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281759 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-run-multus-certs\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281782 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-systemd-units\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281804 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-cni-bin\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281868 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.281915 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-kubernetes\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282105 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8c01759-3d25-4362-922e-dc595a65771e-konnectivity-ca\") pod \"konnectivity-agent-hpg7p\" (UID: \"d8c01759-3d25-4362-922e-dc595a65771e\") " pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282162 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-cnibin\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.282329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282200 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvq7s\" (UniqueName: \"kubernetes.io/projected/34a812d1-d5c3-4506-9892-351debc35566-kube-api-access-wvq7s\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282239 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ebe845f-c3d6-4eed-a05e-88b608cc827f-kubelet-config\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282281 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-os-release\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282306 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-var-lib-cni-multus\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282332 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b97d4e52-96f4-4464-ad58-a03904d77083-multus-daemon-config\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282354 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-tmp\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282379 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-device-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282404 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/158c7ddf-1b84-497b-930e-f5df14579893-host-slash\") pod \"iptables-alerter-cc6lv\" (UID: \"158c7ddf-1b84-497b-930e-f5df14579893\") " pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282427 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-etc-openvswitch\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282469 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-node-log\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282495 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282520 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/721cdb29-0d68-4127-b176-0f7d020a53c4-ovnkube-script-lib\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282544 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpmxs\" (UniqueName: \"kubernetes.io/projected/379c825d-878f-4066-841c-084c71477f94-kube-api-access-zpmxs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282582 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-registration-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282613 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/158c7ddf-1b84-497b-930e-f5df14579893-iptables-alerter-script\") pod \"iptables-alerter-cc6lv\" (UID: \"158c7ddf-1b84-497b-930e-f5df14579893\") " pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282637 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-system-cni-dir\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.283128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282659 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-log-socket\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.284015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282683 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2p6\" (UniqueName: \"kubernetes.io/projected/721cdb29-0d68-4127-b176-0f7d020a53c4-kube-api-access-xb2p6\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.284015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282708 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a1dbd3-0eba-4141-a243-bfb4b7da0740-host\") pod \"node-ca-v5hwn\" (UID: \"e9a1dbd3-0eba-4141-a243-bfb4b7da0740\") " pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.284015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282732 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4lm\" (UniqueName: \"kubernetes.io/projected/b58c15d4-68ef-43b8-bf05-7a08b987306d-kube-api-access-9b4lm\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.284015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282756 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw28f\" (UniqueName: \"kubernetes.io/projected/19853864-4d88-4abe-8485-aa693e06ab08-kube-api-access-kw28f\") pod \"node-resolver-w5jp9\" (UID: \"19853864-4d88-4abe-8485-aa693e06ab08\") " pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.284015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282779 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-sys-fs\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.284015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282803 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m24vk\" (UniqueName: \"kubernetes.io/projected/158c7ddf-1b84-497b-930e-f5df14579893-kube-api-access-m24vk\") pod \"iptables-alerter-cc6lv\" (UID: \"158c7ddf-1b84-497b-930e-f5df14579893\") " pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.284015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282830 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-run-k8s-cni-cncf-io\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.284015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.282854 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-var-lib-cni-bin\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.288185 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.288166 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 14:01:09.307855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.307835 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hjq4r" Jul 29 14:01:09.316120 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.316100 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hjq4r" Jul 29 14:01:09.370465 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.370430 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 14:01:09.379883 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.379826 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" event={"ID":"e66e6a37cab01c0d262e04ba35d56b5e","Type":"ContainerStarted","Data":"902a3172096a64abc40e2253fe41f54362b09ecacd2db9d2371c825f7afd02c3"} Jul 29 14:01:09.381321 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.381292 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal" event={"ID":"5cca166521caabdf057abdac110b1500","Type":"ContainerStarted","Data":"8ea20a2cb2e29b8e99a9997237f84baef69c78690237076692f8a29244f0c066"} Jul 29 14:01:09.383561 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383525 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ebe845f-c3d6-4eed-a05e-88b608cc827f-kubelet-config\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.383672 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383563 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-os-release\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.383672 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383592 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-var-lib-cni-multus\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.383672 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383635 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b97d4e52-96f4-4464-ad58-a03904d77083-multus-daemon-config\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.383672 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383662 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-tmp\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.383892 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383688 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-device-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.383892 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383722 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/158c7ddf-1b84-497b-930e-f5df14579893-host-slash\") pod \"iptables-alerter-cc6lv\" (UID: \"158c7ddf-1b84-497b-930e-f5df14579893\") " pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.383892 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383746 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-etc-openvswitch\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.383892 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383773 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-node-log\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.383892 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383800 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.383892 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383851 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/721cdb29-0d68-4127-b176-0f7d020a53c4-ovnkube-script-lib\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.383892 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383878 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpmxs\" (UniqueName: \"kubernetes.io/projected/379c825d-878f-4066-841c-084c71477f94-kube-api-access-zpmxs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383904 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-registration-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383929 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/158c7ddf-1b84-497b-930e-f5df14579893-iptables-alerter-script\") pod \"iptables-alerter-cc6lv\" (UID: \"158c7ddf-1b84-497b-930e-f5df14579893\") " pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383954 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-system-cni-dir\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383978 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-log-socket\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.383997 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2p6\" (UniqueName: \"kubernetes.io/projected/721cdb29-0d68-4127-b176-0f7d020a53c4-kube-api-access-xb2p6\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384013 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a1dbd3-0eba-4141-a243-bfb4b7da0740-host\") pod \"node-ca-v5hwn\" (UID: \"e9a1dbd3-0eba-4141-a243-bfb4b7da0740\") " pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384031 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4lm\" (UniqueName: \"kubernetes.io/projected/b58c15d4-68ef-43b8-bf05-7a08b987306d-kube-api-access-9b4lm\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384048 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw28f\" (UniqueName: \"kubernetes.io/projected/19853864-4d88-4abe-8485-aa693e06ab08-kube-api-access-kw28f\") pod \"node-resolver-w5jp9\" (UID: \"19853864-4d88-4abe-8485-aa693e06ab08\") " pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384065 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-sys-fs\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384089 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24vk\" (UniqueName: \"kubernetes.io/projected/158c7ddf-1b84-497b-930e-f5df14579893-kube-api-access-m24vk\") pod \"iptables-alerter-cc6lv\" (UID: \"158c7ddf-1b84-497b-930e-f5df14579893\") " pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384109 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-run-k8s-cni-cncf-io\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384125 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-var-lib-cni-bin\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.384148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384140 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-run-netns\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-run-netns\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384176 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-slash\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384190 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-run-openvswitch\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384206 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/721cdb29-0d68-4127-b176-0f7d020a53c4-ovnkube-config\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384228 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8c01759-3d25-4362-922e-dc595a65771e-agent-certs\") pod \"konnectivity-agent-hpg7p\" (UID: \"d8c01759-3d25-4362-922e-dc595a65771e\") " pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384245 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-modprobe-d\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384260 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-sysctl-conf\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384275 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-systemd\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384290 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-host\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384308 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384325 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384342 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrxm\" (UniqueName: \"kubernetes.io/projected/e9a1dbd3-0eba-4141-a243-bfb4b7da0740-kube-api-access-jxrxm\") pod \"node-ca-v5hwn\" (UID: \"e9a1dbd3-0eba-4141-a243-bfb4b7da0740\") " pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384357 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-sys\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384379 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-lib-modules\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384395 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19853864-4d88-4abe-8485-aa693e06ab08-tmp-dir\") pod \"node-resolver-w5jp9\" (UID: \"19853864-4d88-4abe-8485-aa693e06ab08\") " pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384411 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-run-systemd\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384450 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-var-lib-openvswitch\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.384574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384470 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-run-ovn\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384486 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/721cdb29-0d68-4127-b176-0f7d020a53c4-env-overrides\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384516 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-sysctl-d\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384533 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384550 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-system-cni-dir\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384566 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/721cdb29-0d68-4127-b176-0f7d020a53c4-ovn-node-metrics-cert\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384585 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9a1dbd3-0eba-4141-a243-bfb4b7da0740-serviceca\") pod \"node-ca-v5hwn\" (UID: \"e9a1dbd3-0eba-4141-a243-bfb4b7da0740\") " pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384600 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ebe845f-c3d6-4eed-a05e-88b608cc827f-dbus\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384660 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34a812d1-d5c3-4506-9892-351debc35566-cni-binary-copy\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384693 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34a812d1-d5c3-4506-9892-351debc35566-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384723 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-multus-cni-dir\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384748 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-cnibin\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384774 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-multus-socket-dir-parent\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-var-lib-kubelet\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384828 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-os-release\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384846 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b97d4e52-96f4-4464-ad58-a03904d77083-cni-binary-copy\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384862 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-hostroot\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384886 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-multus-conf-dir\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384915 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-kubelet\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384932 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-var-lib-kubelet\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384957 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-tuned\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.384984 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19853864-4d88-4abe-8485-aa693e06ab08-hosts-file\") pod \"node-resolver-w5jp9\" (UID: \"19853864-4d88-4abe-8485-aa693e06ab08\") " pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385010 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-socket-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385036 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385062 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-run\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385087 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctv6\" (UniqueName: \"kubernetes.io/projected/b97d4e52-96f4-4464-ad58-a03904d77083-kube-api-access-dctv6\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385147 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-cni-netd\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385180 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385208 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-sysconfig\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385235 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385267 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34a812d1-d5c3-4506-9892-351debc35566-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385293 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-etc-kubernetes\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385313 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twk52\" (UniqueName: \"kubernetes.io/projected/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-kube-api-access-twk52\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385330 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-run-multus-certs\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.385998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385348 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-systemd-units\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385365 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-cni-bin\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385389 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-kubernetes\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385424 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8c01759-3d25-4362-922e-dc595a65771e-konnectivity-ca\") pod \"konnectivity-agent-hpg7p\" (UID: \"d8c01759-3d25-4362-922e-dc595a65771e\") " pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385441 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-cnibin\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385459 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvq7s\" (UniqueName: \"kubernetes.io/projected/34a812d1-d5c3-4506-9892-351debc35566-kube-api-access-wvq7s\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385832 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ebe845f-c3d6-4eed-a05e-88b608cc827f-kubelet-config\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385896 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-os-release\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.385935 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-var-lib-cni-multus\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.386647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.386397 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b97d4e52-96f4-4464-ad58-a03904d77083-multus-daemon-config\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.387119 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.386676 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 14:01:09.387119 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387086 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/721cdb29-0d68-4127-b176-0f7d020a53c4-env-overrides\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.387203 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387167 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-device-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.387242 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387203 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/158c7ddf-1b84-497b-930e-f5df14579893-host-slash\") pod \"iptables-alerter-cc6lv\" (UID: \"158c7ddf-1b84-497b-930e-f5df14579893\") " pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.387242 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387235 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-etc-openvswitch\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.387311 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387261 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-node-log\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.387352 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387315 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387560 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-multus-cni-dir\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387723 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-sysctl-d\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387781 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19853864-4d88-4abe-8485-aa693e06ab08-hosts-file\") pod \"node-resolver-w5jp9\" (UID: \"19853864-4d88-4abe-8485-aa693e06ab08\") " pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387786 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387828 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-system-cni-dir\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387830 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/721cdb29-0d68-4127-b176-0f7d020a53c4-ovnkube-script-lib\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387904 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-socket-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387937 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.387976 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-run\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388120 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-registration-dir\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388206 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-system-cni-dir\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388247 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-cni-netd\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388256 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ebe845f-c3d6-4eed-a05e-88b608cc827f-dbus\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388370 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-sysctl-conf\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388454 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-modprobe-d\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388506 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-systemd\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388557 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-host\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.390403 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.388676 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388726 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9a1dbd3-0eba-4141-a243-bfb4b7da0740-serviceca\") pod \"node-ca-v5hwn\" (UID: \"e9a1dbd3-0eba-4141-a243-bfb4b7da0740\") " pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.388746 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret podName:6ebe845f-c3d6-4eed-a05e-88b608cc827f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:09.888727097 +0000 UTC m=+3.090228250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret") pod "global-pull-secret-syncer-2vm4f" (UID: "6ebe845f-c3d6-4eed-a05e-88b608cc827f") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388907 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.388966 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/158c7ddf-1b84-497b-930e-f5df14579893-iptables-alerter-script\") pod \"iptables-alerter-cc6lv\" (UID: \"158c7ddf-1b84-497b-930e-f5df14579893\") " pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389173 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-run-multus-certs\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389200 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34a812d1-d5c3-4506-9892-351debc35566-cni-binary-copy\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389262 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-log-socket\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389321 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-sysconfig\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389466 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-systemd-units\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389490 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19853864-4d88-4abe-8485-aa693e06ab08-tmp-dir\") pod \"node-resolver-w5jp9\" (UID: \"19853864-4d88-4abe-8485-aa693e06ab08\") " pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389520 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-cni-bin\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389554 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a1dbd3-0eba-4141-a243-bfb4b7da0740-host\") pod \"node-ca-v5hwn\" (UID: \"e9a1dbd3-0eba-4141-a243-bfb4b7da0740\") " pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.389597 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389651 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-lib-modules\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.389661 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs podName:379c825d-878f-4066-841c-084c71477f94 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:09.889645954 +0000 UTC m=+3.091147108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs") pod "network-metrics-daemon-vfv5m" (UID: "379c825d-878f-4066-841c-084c71477f94") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389705 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-run-systemd\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.391612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389705 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-kubernetes\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389178 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-sys\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.389985 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-var-lib-openvswitch\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390001 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-run-ovn\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390033 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-hostroot\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390043 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-run-netns\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390055 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34a812d1-d5c3-4506-9892-351debc35566-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390070 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34a812d1-d5c3-4506-9892-351debc35566-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390097 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-sys-fs\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-var-lib-kubelet\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390135 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-run-k8s-cni-cncf-io\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390155 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-cnibin\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390184 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-os-release\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390203 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-multus-socket-dir-parent\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390245 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-run-openvswitch\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390258 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b58c15d4-68ef-43b8-bf05-7a08b987306d-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390279 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-var-lib-cni-bin\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390294 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34a812d1-d5c3-4506-9892-351debc35566-cnibin\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.392540 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390301 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8c01759-3d25-4362-922e-dc595a65771e-konnectivity-ca\") pod \"konnectivity-agent-hpg7p\" (UID: \"d8c01759-3d25-4362-922e-dc595a65771e\") " pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390314 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-slash\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390326 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-multus-conf-dir\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390329 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-tmp\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390343 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/721cdb29-0d68-4127-b176-0f7d020a53c4-host-kubelet\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390348 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-host-run-netns\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390366 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b97d4e52-96f4-4464-ad58-a03904d77083-etc-kubernetes\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390383 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-var-lib-kubelet\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390449 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b97d4e52-96f4-4464-ad58-a03904d77083-cni-binary-copy\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390730 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/721cdb29-0d68-4127-b176-0f7d020a53c4-ovn-node-metrics-cert\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.390958 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/721cdb29-0d68-4127-b176-0f7d020a53c4-ovnkube-config\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.393477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.393293 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-etc-tuned\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.394126 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.394105 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8c01759-3d25-4362-922e-dc595a65771e-agent-certs\") pod \"konnectivity-agent-hpg7p\" (UID: \"d8c01759-3d25-4362-922e-dc595a65771e\") " pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:09.394735 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.394715 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvq7s\" (UniqueName: \"kubernetes.io/projected/34a812d1-d5c3-4506-9892-351debc35566-kube-api-access-wvq7s\") pod \"multus-additional-cni-plugins-rrvvf\" (UID: \"34a812d1-d5c3-4506-9892-351debc35566\") " pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.401489 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.401463 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:09.401609 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.401495 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:09.401609 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.401509 2560 projected.go:196] Error preparing data for projected volume kube-api-access-7kl8n for pod openshift-network-diagnostics/network-check-target-8qt8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:09.401727 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.401631 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n podName:7863615c-db3b-4db2-976d-22e559411059 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:09.901594612 +0000 UTC m=+3.103095785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7kl8n" (UniqueName: "kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n") pod "network-check-target-8qt8j" (UID: "7863615c-db3b-4db2-976d-22e559411059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:09.404356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.403776 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twk52\" (UniqueName: \"kubernetes.io/projected/1f8a83db-61cb-4c54-9dec-94ec74bdcb9a-kube-api-access-twk52\") pod \"tuned-hthjh\" (UID: \"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.404356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.404291 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctv6\" (UniqueName: \"kubernetes.io/projected/b97d4e52-96f4-4464-ad58-a03904d77083-kube-api-access-dctv6\") pod \"multus-r7xf6\" (UID: \"b97d4e52-96f4-4464-ad58-a03904d77083\") " pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.404356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.404310 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24vk\" (UniqueName: \"kubernetes.io/projected/158c7ddf-1b84-497b-930e-f5df14579893-kube-api-access-m24vk\") pod \"iptables-alerter-cc6lv\" (UID: \"158c7ddf-1b84-497b-930e-f5df14579893\") " pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.405246 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.405222 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw28f\" (UniqueName: \"kubernetes.io/projected/19853864-4d88-4abe-8485-aa693e06ab08-kube-api-access-kw28f\") pod \"node-resolver-w5jp9\" (UID: \"19853864-4d88-4abe-8485-aa693e06ab08\") " pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.405744 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.405718 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpmxs\" (UniqueName: \"kubernetes.io/projected/379c825d-878f-4066-841c-084c71477f94-kube-api-access-zpmxs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:09.406271 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.406247 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrxm\" (UniqueName: \"kubernetes.io/projected/e9a1dbd3-0eba-4141-a243-bfb4b7da0740-kube-api-access-jxrxm\") pod \"node-ca-v5hwn\" (UID: \"e9a1dbd3-0eba-4141-a243-bfb4b7da0740\") " pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.406647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.406624 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2p6\" (UniqueName: \"kubernetes.io/projected/721cdb29-0d68-4127-b176-0f7d020a53c4-kube-api-access-xb2p6\") pod \"ovnkube-node-plvns\" (UID: \"721cdb29-0d68-4127-b176-0f7d020a53c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.408474 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.408450 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4lm\" (UniqueName: \"kubernetes.io/projected/b58c15d4-68ef-43b8-bf05-7a08b987306d-kube-api-access-9b4lm\") pod \"aws-ebs-csi-driver-node-sjk69\" (UID: \"b58c15d4-68ef-43b8-bf05-7a08b987306d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.564844 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.564757 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" Jul 29 14:01:09.573109 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.573081 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cc6lv" Jul 29 14:01:09.582773 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.582739 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v5hwn" Jul 29 14:01:09.587705 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.587669 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" Jul 29 14:01:09.594684 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.594657 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hthjh" Jul 29 14:01:09.602647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.602620 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w5jp9" Jul 29 14:01:09.608633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.608611 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7xf6" Jul 29 14:01:09.616394 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.616229 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:09.622619 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.622592 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:09.723675 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.723452 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:09.890178 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.890066 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:09.890178 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.890161 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:09.890425 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.890255 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:09.890425 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.890282 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:09.890425 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.890342 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs podName:379c825d-878f-4066-841c-084c71477f94 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:10.890320442 +0000 UTC m=+4.091821595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs") pod "network-metrics-daemon-vfv5m" (UID: "379c825d-878f-4066-841c-084c71477f94") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:09.890425 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.890359 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret podName:6ebe845f-c3d6-4eed-a05e-88b608cc827f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:10.890350959 +0000 UTC m=+4.091852110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret") pod "global-pull-secret-syncer-2vm4f" (UID: "6ebe845f-c3d6-4eed-a05e-88b608cc827f") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:09.991556 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:09.991495 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:09.991738 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.991716 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:09.991811 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.991743 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:09.991811 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.991758 2560 projected.go:196] Error preparing data for projected volume kube-api-access-7kl8n for pod openshift-network-diagnostics/network-check-target-8qt8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:09.991898 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:09.991833 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n podName:7863615c-db3b-4db2-976d-22e559411059 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:10.991812084 +0000 UTC m=+4.193313251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7kl8n" (UniqueName: "kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n") pod "network-check-target-8qt8j" (UID: "7863615c-db3b-4db2-976d-22e559411059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:10.319391 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:10.319350 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 13:56:09 +0000 UTC" deadline="2028-03-18 04:36:04.890593752 +0000 UTC" Jul 29 14:01:10.319391 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:10.319384 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14342h34m54.571212718s" Jul 29 14:01:10.374984 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:10.374951 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:10.375164 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:10.375081 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:10.644784 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:10.644752 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8a83db_61cb_4c54_9dec_94ec74bdcb9a.slice/crio-606d00dcecc787b61a91f8b9a8d5e888028b1d59c985f12e4fbb689742adb265 WatchSource:0}: Error finding container 606d00dcecc787b61a91f8b9a8d5e888028b1d59c985f12e4fbb689742adb265: Status 404 returned error can't find the container with id 606d00dcecc787b61a91f8b9a8d5e888028b1d59c985f12e4fbb689742adb265 Jul 29 14:01:10.646073 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:10.646047 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a812d1_d5c3_4506_9892_351debc35566.slice/crio-f50afa04970496ad0c5abaaec453d91bffe2f08e95c1b366c7729d2c64f256aa WatchSource:0}: Error finding container f50afa04970496ad0c5abaaec453d91bffe2f08e95c1b366c7729d2c64f256aa: Status 404 returned error can't find the container with id f50afa04970496ad0c5abaaec453d91bffe2f08e95c1b366c7729d2c64f256aa Jul 29 14:01:10.646552 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:10.646503 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158c7ddf_1b84_497b_930e_f5df14579893.slice/crio-c21088a3b7fc87e5aaac9a91b10006a929d16ac70091448168d34ffe497a04fd WatchSource:0}: Error finding container c21088a3b7fc87e5aaac9a91b10006a929d16ac70091448168d34ffe497a04fd: Status 404 returned error can't find the container with id c21088a3b7fc87e5aaac9a91b10006a929d16ac70091448168d34ffe497a04fd Jul 29 14:01:10.648281 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:10.648185 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58c15d4_68ef_43b8_bf05_7a08b987306d.slice/crio-426fb570bcaf84436c4ca94c33f79747039d167389588142794f672c995cc55e WatchSource:0}: Error finding container 426fb570bcaf84436c4ca94c33f79747039d167389588142794f672c995cc55e: Status 404 returned error can't find the container with id 426fb570bcaf84436c4ca94c33f79747039d167389588142794f672c995cc55e Jul 29 14:01:10.653452 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:10.653135 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a1dbd3_0eba_4141_a243_bfb4b7da0740.slice/crio-49098228d0c0f876af9822d67094c4e90f7d2adb69fa016eb48c1d2555dfb1bd WatchSource:0}: Error finding container 49098228d0c0f876af9822d67094c4e90f7d2adb69fa016eb48c1d2555dfb1bd: Status 404 returned error can't find the container with id 49098228d0c0f876af9822d67094c4e90f7d2adb69fa016eb48c1d2555dfb1bd Jul 29 14:01:10.654188 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:10.654107 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97d4e52_96f4_4464_ad58_a03904d77083.slice/crio-4ce590f9db4f46b631d715b74aed01c1ae1ea74b77b974cd08af01b63a58dae6 WatchSource:0}: Error finding container 4ce590f9db4f46b631d715b74aed01c1ae1ea74b77b974cd08af01b63a58dae6: Status 404 returned error can't find the container with id 4ce590f9db4f46b631d715b74aed01c1ae1ea74b77b974cd08af01b63a58dae6 Jul 29 14:01:10.655774 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:10.655390 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c01759_3d25_4362_922e_dc595a65771e.slice/crio-a7f71dbd5dbbd9eebf6d1fa6a37588cdc520aa1917c1d6ca5a2461d6c6114925 WatchSource:0}: Error finding container a7f71dbd5dbbd9eebf6d1fa6a37588cdc520aa1917c1d6ca5a2461d6c6114925: Status 404 returned error can't find the container with id a7f71dbd5dbbd9eebf6d1fa6a37588cdc520aa1917c1d6ca5a2461d6c6114925 Jul 29 14:01:10.656517 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:10.656491 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19853864_4d88_4abe_8485_aa693e06ab08.slice/crio-dadb6dd23150622cc2607f8bb19ed8138e7153e8ed7b1910f5ecd4f398283aa1 WatchSource:0}: Error finding container dadb6dd23150622cc2607f8bb19ed8138e7153e8ed7b1910f5ecd4f398283aa1: Status 404 returned error can't find the container with id dadb6dd23150622cc2607f8bb19ed8138e7153e8ed7b1910f5ecd4f398283aa1 Jul 29 14:01:10.659646 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:10.659614 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721cdb29_0d68_4127_b176_0f7d020a53c4.slice/crio-a2bb9f9e1ba107b5aa138df257583da8d7017fcb28eac449d7d285b8556fb7b7 WatchSource:0}: Error finding container a2bb9f9e1ba107b5aa138df257583da8d7017fcb28eac449d7d285b8556fb7b7: Status 404 returned error can't find the container with id a2bb9f9e1ba107b5aa138df257583da8d7017fcb28eac449d7d285b8556fb7b7 Jul 29 14:01:10.898602 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:10.898422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:10.898716 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:10.898604 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:10.898716 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:10.898668 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret podName:6ebe845f-c3d6-4eed-a05e-88b608cc827f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:12.898651761 +0000 UTC m=+6.100152912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret") pod "global-pull-secret-syncer-2vm4f" (UID: "6ebe845f-c3d6-4eed-a05e-88b608cc827f") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:10.898716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:10.898689 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:10.898862 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:10.898765 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:10.898862 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:10.898799 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs podName:379c825d-878f-4066-841c-084c71477f94 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:12.898788698 +0000 UTC m=+6.100289850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs") pod "network-metrics-daemon-vfv5m" (UID: "379c825d-878f-4066-841c-084c71477f94") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:10.999292 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:10.999260 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:10.999423 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:10.999392 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:10.999423 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:10.999405 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:10.999423 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:10.999415 2560 projected.go:196] Error preparing data for projected volume kube-api-access-7kl8n for pod openshift-network-diagnostics/network-check-target-8qt8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:10.999534 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:10.999465 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n podName:7863615c-db3b-4db2-976d-22e559411059 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:12.99945191 +0000 UTC m=+6.200953062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7kl8n" (UniqueName: "kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n") pod "network-check-target-8qt8j" (UID: "7863615c-db3b-4db2-976d-22e559411059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:11.179795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.179762 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 14:01:11.320388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.320310 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 13:56:09 +0000 UTC" deadline="2028-04-09 04:18:40.498869778 +0000 UTC" Jul 29 14:01:11.320388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.320353 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14870h17m29.178521708s" Jul 29 14:01:11.375346 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.374842 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:11.375346 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:11.374981 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:11.375346 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.375021 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:11.375346 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:11.375143 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:11.390793 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.390000 2560 generic.go:356] "Generic (PLEG): container finished" podID="e66e6a37cab01c0d262e04ba35d56b5e" containerID="73af7eb711a4349edf870d72e78cea5d35d5764c125ef41ed62e632e3492aebd" exitCode=0 Jul 29 14:01:11.390793 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.390566 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" event={"ID":"e66e6a37cab01c0d262e04ba35d56b5e","Type":"ContainerDied","Data":"73af7eb711a4349edf870d72e78cea5d35d5764c125ef41ed62e632e3492aebd"} Jul 29 14:01:11.397762 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.397645 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w5jp9" event={"ID":"19853864-4d88-4abe-8485-aa693e06ab08","Type":"ContainerStarted","Data":"dadb6dd23150622cc2607f8bb19ed8138e7153e8ed7b1910f5ecd4f398283aa1"} Jul 29 14:01:11.399975 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.399945 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hpg7p" event={"ID":"d8c01759-3d25-4362-922e-dc595a65771e","Type":"ContainerStarted","Data":"a7f71dbd5dbbd9eebf6d1fa6a37588cdc520aa1917c1d6ca5a2461d6c6114925"} Jul 29 14:01:11.402349 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.402322 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7xf6" event={"ID":"b97d4e52-96f4-4464-ad58-a03904d77083","Type":"ContainerStarted","Data":"4ce590f9db4f46b631d715b74aed01c1ae1ea74b77b974cd08af01b63a58dae6"} Jul 29 14:01:11.410765 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.410738 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cc6lv" event={"ID":"158c7ddf-1b84-497b-930e-f5df14579893","Type":"ContainerStarted","Data":"c21088a3b7fc87e5aaac9a91b10006a929d16ac70091448168d34ffe497a04fd"} Jul 29 14:01:11.414360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.414001 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal" event={"ID":"5cca166521caabdf057abdac110b1500","Type":"ContainerStarted","Data":"eee30b8f24b3da64b61a780f09c767311d8f08f3a8dd384f5bd55782dfe6d495"} Jul 29 14:01:11.416188 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.416162 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerStarted","Data":"a2bb9f9e1ba107b5aa138df257583da8d7017fcb28eac449d7d285b8556fb7b7"} Jul 29 14:01:11.423210 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.423178 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v5hwn" event={"ID":"e9a1dbd3-0eba-4141-a243-bfb4b7da0740","Type":"ContainerStarted","Data":"49098228d0c0f876af9822d67094c4e90f7d2adb69fa016eb48c1d2555dfb1bd"} Jul 29 14:01:11.424786 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.424759 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" event={"ID":"b58c15d4-68ef-43b8-bf05-7a08b987306d","Type":"ContainerStarted","Data":"426fb570bcaf84436c4ca94c33f79747039d167389588142794f672c995cc55e"} Jul 29 14:01:11.426222 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.426162 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" event={"ID":"34a812d1-d5c3-4506-9892-351debc35566","Type":"ContainerStarted","Data":"f50afa04970496ad0c5abaaec453d91bffe2f08e95c1b366c7729d2c64f256aa"} Jul 29 14:01:11.430338 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:11.430253 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hthjh" event={"ID":"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a","Type":"ContainerStarted","Data":"606d00dcecc787b61a91f8b9a8d5e888028b1d59c985f12e4fbb689742adb265"} Jul 29 14:01:12.374970 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:12.374932 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:12.375529 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:12.375087 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:12.438512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:12.438471 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" event={"ID":"e66e6a37cab01c0d262e04ba35d56b5e","Type":"ContainerStarted","Data":"7e7bc119cfc5d27cbe44b3f6ee07cd543671e93d40fffac6b92225e566fae448"} Jul 29 14:01:12.480367 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:12.480291 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-155.ec2.internal" podStartSLOduration=4.48027001 podStartE2EDuration="4.48027001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:11.432523055 +0000 UTC m=+4.634024234" watchObservedRunningTime="2026-07-29 14:01:12.48027001 +0000 UTC m=+5.681771187" Jul 29 14:01:12.916752 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:12.916716 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:12.916957 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:12.916781 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:12.916957 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:12.916877 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:12.916957 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:12.916938 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs podName:379c825d-878f-4066-841c-084c71477f94 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:16.916919919 +0000 UTC m=+10.118421071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs") pod "network-metrics-daemon-vfv5m" (UID: "379c825d-878f-4066-841c-084c71477f94") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:12.917307 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:12.917287 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:12.917380 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:12.917349 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret podName:6ebe845f-c3d6-4eed-a05e-88b608cc827f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:16.917331615 +0000 UTC m=+10.118832768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret") pod "global-pull-secret-syncer-2vm4f" (UID: "6ebe845f-c3d6-4eed-a05e-88b608cc827f") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:13.018031 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:13.017951 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:13.018200 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:13.018135 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:13.018200 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:13.018157 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:13.018200 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:13.018167 2560 projected.go:196] Error preparing data for projected volume kube-api-access-7kl8n for pod openshift-network-diagnostics/network-check-target-8qt8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:13.018385 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:13.018220 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n podName:7863615c-db3b-4db2-976d-22e559411059 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:17.018199607 +0000 UTC m=+10.219700774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7kl8n" (UniqueName: "kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n") pod "network-check-target-8qt8j" (UID: "7863615c-db3b-4db2-976d-22e559411059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:13.374798 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:13.374760 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:13.374984 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:13.374958 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:13.375364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:13.374760 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:13.375364 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:13.375076 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:14.374553 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:14.374505 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:14.374739 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:14.374658 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:15.374654 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:15.374626 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:15.375109 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:15.374722 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:15.375642 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:15.375619 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:15.375768 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:15.375746 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:16.375212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:16.374669 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:16.375212 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:16.374826 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:16.953009 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:16.952963 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:16.953196 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:16.953026 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:16.953196 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:16.953141 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:16.953318 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:16.953203 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret podName:6ebe845f-c3d6-4eed-a05e-88b608cc827f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:24.953185009 +0000 UTC m=+18.154686175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret") pod "global-pull-secret-syncer-2vm4f" (UID: "6ebe845f-c3d6-4eed-a05e-88b608cc827f") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:16.953635 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:16.953612 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:16.953725 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:16.953673 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs podName:379c825d-878f-4066-841c-084c71477f94 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:24.953658139 +0000 UTC m=+18.155159291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs") pod "network-metrics-daemon-vfv5m" (UID: "379c825d-878f-4066-841c-084c71477f94") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:17.054972 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:17.054057 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:17.054972 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:17.054237 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:17.054972 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:17.054259 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:17.054972 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:17.054272 2560 projected.go:196] Error preparing data for projected volume kube-api-access-7kl8n for pod openshift-network-diagnostics/network-check-target-8qt8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:17.054972 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:17.054333 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n podName:7863615c-db3b-4db2-976d-22e559411059 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:25.054312739 +0000 UTC m=+18.255813910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7kl8n" (UniqueName: "kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n") pod "network-check-target-8qt8j" (UID: "7863615c-db3b-4db2-976d-22e559411059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:17.375089 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:17.375010 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:17.375240 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:17.375110 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:17.375240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:17.375210 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:17.375665 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:17.375316 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:18.374935 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:18.374902 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:18.375118 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:18.375043 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:19.373979 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:19.373879 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:19.373979 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:19.373879 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:19.374466 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:19.373983 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:19.374466 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:19.374048 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:20.374122 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:20.374083 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:20.374561 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:20.374240 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:21.374326 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.374083 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:21.374782 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.374084 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:21.374782 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:21.374431 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:21.374782 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:21.374505 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:21.461524 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.461491 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v5hwn" event={"ID":"e9a1dbd3-0eba-4141-a243-bfb4b7da0740","Type":"ContainerStarted","Data":"a2ab7c515336e6be2a246d1a13179955d87ea3df47c61378936c24a09cc55c7a"} Jul 29 14:01:21.463257 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.463231 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" event={"ID":"b58c15d4-68ef-43b8-bf05-7a08b987306d","Type":"ContainerStarted","Data":"78b4bfb53f0c1ec6cfaf5d669b86dc05572d3bc96c97541611f69eaf1b4d835a"} Jul 29 14:01:21.465011 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.464982 2560 generic.go:356] "Generic (PLEG): container finished" podID="34a812d1-d5c3-4506-9892-351debc35566" containerID="81939d16187ad1383a7ca625f966a3aafbb24936e039bda97278d84cf27559b8" exitCode=0 Jul 29 14:01:21.465102 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.465060 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" event={"ID":"34a812d1-d5c3-4506-9892-351debc35566","Type":"ContainerDied","Data":"81939d16187ad1383a7ca625f966a3aafbb24936e039bda97278d84cf27559b8"} Jul 29 14:01:21.466520 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.466496 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hthjh" event={"ID":"1f8a83db-61cb-4c54-9dec-94ec74bdcb9a","Type":"ContainerStarted","Data":"4dd1fb96cdc600d1a893a98e826fd9ee0c99d111983ccdbf5499fe4b2a71645f"} Jul 29 14:01:21.468270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.468251 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w5jp9" event={"ID":"19853864-4d88-4abe-8485-aa693e06ab08","Type":"ContainerStarted","Data":"eabd1de2e1af1303f2f6f9ec10fe9847221c55dfe3d18e5dc52910017b99dbf4"} Jul 29 14:01:21.471088 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.471063 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hpg7p" event={"ID":"d8c01759-3d25-4362-922e-dc595a65771e","Type":"ContainerStarted","Data":"379d60c8881873abb5c9840d6a826aadd1f1bf1c1655808abd199a3225fb2d03"} Jul 29 14:01:21.478399 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.478362 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-155.ec2.internal" podStartSLOduration=13.478345702 podStartE2EDuration="13.478345702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:12.480714982 +0000 UTC m=+5.682216162" watchObservedRunningTime="2026-07-29 14:01:21.478345702 +0000 UTC m=+14.679846875" Jul 29 14:01:21.491365 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.491322 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v5hwn" podStartSLOduration=6.067579766 podStartE2EDuration="14.491312594s" totalImagesPullingTime="8.423732828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:21.47801723 +0000 UTC m=+14.679518403" watchObservedRunningTime="2026-07-29 14:01:21.491312594 +0000 UTC m=+14.692813766" Jul 29 14:01:21.491855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.491795 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hpg7p" podStartSLOduration=3.487044562 podStartE2EDuration="13.49178905s" totalImagesPullingTime="10.004744488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:21.491197061 +0000 UTC m=+14.692698240" watchObservedRunningTime="2026-07-29 14:01:21.49178905 +0000 UTC m=+14.693290226" Jul 29 14:01:21.541851 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:21.541800 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hthjh" podStartSLOduration=4.495522823 podStartE2EDuration="14.541781767s" totalImagesPullingTime="10.046258944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:21.528466773 +0000 UTC m=+14.729967945" watchObservedRunningTime="2026-07-29 14:01:21.541781767 +0000 UTC m=+14.743282938" Jul 29 14:01:22.374139 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:22.374047 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:22.374289 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:22.374197 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:22.474859 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:22.474819 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cc6lv" event={"ID":"158c7ddf-1b84-497b-930e-f5df14579893","Type":"ContainerStarted","Data":"a643c47e552bbd8833fe620ed59b85bebab9bc8fb9ca7214e36b462c18a12aec"} Jul 29 14:01:22.490041 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:22.489992 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w5jp9" podStartSLOduration=4.493139922 podStartE2EDuration="14.489978017s" totalImagesPullingTime="9.996838095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:21.541638434 +0000 UTC m=+14.743139608" watchObservedRunningTime="2026-07-29 14:01:22.489978017 +0000 UTC m=+15.691479221" Jul 29 14:01:22.490497 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:22.490470 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cc6lv" podStartSLOduration=5.48294927 podStartE2EDuration="15.490463239s" totalImagesPullingTime="10.007513969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:22.490121616 +0000 UTC m=+15.691622784" watchObservedRunningTime="2026-07-29 14:01:22.490463239 +0000 UTC m=+15.691964410" Jul 29 14:01:23.374405 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:23.374367 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:23.374582 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:23.374495 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:23.374696 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:23.374368 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:23.374820 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:23.374769 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:24.374805 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:24.374773 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:24.375310 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:24.374897 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:25.017512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:25.017479 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:25.017682 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:25.017580 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:25.017682 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.017621 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:25.017682 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.017674 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:25.017778 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.017679 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret podName:6ebe845f-c3d6-4eed-a05e-88b608cc827f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:41.017663745 +0000 UTC m=+34.219164895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret") pod "global-pull-secret-syncer-2vm4f" (UID: "6ebe845f-c3d6-4eed-a05e-88b608cc827f") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:25.017778 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.017730 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs podName:379c825d-878f-4066-841c-084c71477f94 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:41.017714703 +0000 UTC m=+34.219215853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs") pod "network-metrics-daemon-vfv5m" (UID: "379c825d-878f-4066-841c-084c71477f94") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:25.117927 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:25.117901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:25.118083 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.118063 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:25.118083 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.118080 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:25.118173 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.118089 2560 projected.go:196] Error preparing data for projected volume kube-api-access-7kl8n for pod openshift-network-diagnostics/network-check-target-8qt8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:25.118173 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.118142 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n podName:7863615c-db3b-4db2-976d-22e559411059 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:41.118122475 +0000 UTC m=+34.319623630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7kl8n" (UniqueName: "kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n") pod "network-check-target-8qt8j" (UID: "7863615c-db3b-4db2-976d-22e559411059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:25.374325 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:25.374241 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:25.374489 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:25.374254 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:25.374489 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.374377 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:25.374489 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:25.374446 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:25.966551 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:25.966516 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:25.967355 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:25.967331 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:26.374818 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:26.374741 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:26.374954 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:26.374891 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:27.375049 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:27.375018 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:27.375623 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:27.375126 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:27.375623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:27.375212 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:27.375623 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:27.375317 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:28.374251 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:28.374217 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:28.374422 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:28.374364 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:29.374885 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:29.374638 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:29.375320 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:29.374997 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:29.375320 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:29.374638 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:29.375320 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:29.375156 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:29.492882 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:29.492287 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7xf6" event={"ID":"b97d4e52-96f4-4464-ad58-a03904d77083","Type":"ContainerStarted","Data":"9760c7e711f9ac6782296d3c0fb3e214abc501d123c21b5666b93ea13c793061"} Jul 29 14:01:29.494588 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:29.494533 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:01:29.494839 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:29.494820 2560 generic.go:356] "Generic (PLEG): container finished" podID="721cdb29-0d68-4127-b176-0f7d020a53c4" containerID="c6f7c96d2552f4f56a3e4e187857c4cfed7318dfdd17a1c76af134fea066660c" exitCode=1 Jul 29 14:01:29.494921 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:29.494857 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerStarted","Data":"215c3e393626815f907084a28742c1c7f6bc3a8c9ddc7674bd133e2588381882"} Jul 29 14:01:29.494981 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:29.494933 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerDied","Data":"c6f7c96d2552f4f56a3e4e187857c4cfed7318dfdd17a1c76af134fea066660c"} Jul 29 14:01:29.495039 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:29.494986 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerStarted","Data":"d66fb799cd9d9de8e779778a6ec8cbf53f0ea205c572dd8acb61d597a401844a"} Jul 29 14:01:29.511978 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:29.511935 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r7xf6" podStartSLOduration=3.004999354 podStartE2EDuration="21.511924139s" totalImagesPullingTime="18.506924785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:29.511802098 +0000 UTC m=+22.713303272" watchObservedRunningTime="2026-07-29 14:01:29.511924139 +0000 UTC m=+22.713425311" Jul 29 14:01:30.263347 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.263190 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 14:01:30.336217 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.336111 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T14:01:30.263338024Z","UUID":"287f1345-e14c-4314-8914-4464b73e0e30","Handler":null,"Name":"","Endpoint":""} Jul 29 14:01:30.337947 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.337921 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 14:01:30.338075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.337972 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 14:01:30.374696 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.374629 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:30.374837 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:30.374736 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:30.501900 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.501875 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:01:30.502336 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.502311 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerStarted","Data":"1803c44c953cc1cc59004bb4b501ff752cf1b75ba1df97ed1ad2f26719ba61a8"} Jul 29 14:01:30.502413 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.502376 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerStarted","Data":"5d8a5d18a5cde2d42f45705c24cccfd3b316c42fb6ad88794f4f522b111820eb"} Jul 29 14:01:30.502453 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.502417 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerStarted","Data":"f6428d168fe75d6d0a4234e78f324a06ca0b6e64a6552f563a745266ca8e15ae"} Jul 29 14:01:30.504401 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:30.504373 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" event={"ID":"b58c15d4-68ef-43b8-bf05-7a08b987306d","Type":"ContainerStarted","Data":"591ead5db2f14be34145badc529f0f5aa2cbe6d35b9f960a0dd1b92849dce1fb"} Jul 29 14:01:31.374388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:31.374362 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:31.374594 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:31.374465 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:31.374594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:31.374525 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:31.374714 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:31.374653 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:32.374405 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:32.374373 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:32.374806 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:32.374492 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:33.374197 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:33.374114 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:33.374197 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:33.374134 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:33.374410 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:33.374212 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:33.374410 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:33.374337 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:33.514551 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:33.514509 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:01:33.514885 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:33.514861 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerStarted","Data":"942df8903d8e41a7b8d9a3a852b9b995bbf1944f46b1951d82c6ed58f527e2c0"} Jul 29 14:01:33.516757 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:33.516733 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" event={"ID":"b58c15d4-68ef-43b8-bf05-7a08b987306d","Type":"ContainerStarted","Data":"bf1ed1be2c9be8ad7763cf4ce8691736905dcb52206edd1d29a00026621e63fd"} Jul 29 14:01:33.518400 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:33.518379 2560 generic.go:356] "Generic (PLEG): container finished" podID="34a812d1-d5c3-4506-9892-351debc35566" containerID="391fafc966c67189423fe860d56b672981ceb3f0d42c74c6a73eefbea11d8922" exitCode=0 Jul 29 14:01:33.518459 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:33.518411 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" event={"ID":"34a812d1-d5c3-4506-9892-351debc35566","Type":"ContainerDied","Data":"391fafc966c67189423fe860d56b672981ceb3f0d42c74c6a73eefbea11d8922"} Jul 29 14:01:33.534016 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:33.533977 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjk69" podStartSLOduration=4.504295758 podStartE2EDuration="26.533965231s" totalImagesPullingTime="22.029669473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:33.532696371 +0000 UTC m=+26.734197555" watchObservedRunningTime="2026-07-29 14:01:33.533965231 +0000 UTC m=+26.735466405" Jul 29 14:01:34.344598 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:34.344568 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:34.344779 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:34.344677 2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 29 14:01:34.345100 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:34.345086 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hpg7p" Jul 29 14:01:34.374813 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:34.374792 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:34.375119 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:34.374874 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:35.374199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.373973 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:35.374317 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.374006 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:35.374317 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:35.374244 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:35.374317 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:35.374275 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:35.526215 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.526184 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:01:35.526885 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.526586 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerStarted","Data":"36aebbd933483d0c51174760585b04f60170bf28f0ebab92cd106b9d468d17e4"} Jul 29 14:01:35.526885 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.526862 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:35.526978 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.526939 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:35.527122 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.527101 2560 scope.go:117] "RemoveContainer" containerID="c6f7c96d2552f4f56a3e4e187857c4cfed7318dfdd17a1c76af134fea066660c" Jul 29 14:01:35.529301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.529279 2560 generic.go:356] "Generic (PLEG): container finished" podID="34a812d1-d5c3-4506-9892-351debc35566" containerID="2ede717a69ffe44fe48ed84799a3a8e9e7430da1cfce90781cbbd77b90b42fc3" exitCode=0 Jul 29 14:01:35.529387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.529319 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" event={"ID":"34a812d1-d5c3-4506-9892-351debc35566","Type":"ContainerDied","Data":"2ede717a69ffe44fe48ed84799a3a8e9e7430da1cfce90781cbbd77b90b42fc3"} Jul 29 14:01:35.544796 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.544774 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:35.544936 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:35.544923 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:36.374975 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:36.374940 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:36.375164 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:36.375077 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:36.534794 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:36.534769 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:01:36.535199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:36.535109 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" event={"ID":"721cdb29-0d68-4127-b176-0f7d020a53c4","Type":"ContainerStarted","Data":"540088c7b28e41b7dce63fd349e3dbb8f54e0a0031e2665aff6763a45e74ac93"} Jul 29 14:01:36.535274 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:36.535261 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:01:36.560999 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:36.560952 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" podStartSLOduration=10.067626759 podStartE2EDuration="28.560937945s" totalImagesPullingTime="18.493311186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:36.559609246 +0000 UTC m=+29.761110417" watchObservedRunningTime="2026-07-29 14:01:36.560937945 +0000 UTC m=+29.762439117" Jul 29 14:01:37.375878 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:37.375846 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:37.376026 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:37.375966 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:37.376099 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:37.376023 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:37.376139 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:37.376110 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:37.849868 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:37.849834 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2vm4f"] Jul 29 14:01:37.850256 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:37.850029 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:37.850256 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:37.850202 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:37.854162 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:37.854131 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8qt8j"] Jul 29 14:01:37.854328 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:37.854274 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:37.854399 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:37.854372 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:37.854969 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:37.854936 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfv5m"] Jul 29 14:01:37.855093 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:37.855079 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:37.855203 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:37.855186 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:38.542464 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:38.542233 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" event={"ID":"34a812d1-d5c3-4506-9892-351debc35566","Type":"ContainerStarted","Data":"d13adb2c6174fad50c346f9032730d9779434851745e7608d54b583ae45bd533"} Jul 29 14:01:39.374103 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:39.374071 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:39.374616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:39.374071 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:39.374616 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:39.374168 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:39.374616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:39.374175 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:39.374616 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:39.374256 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:39.374616 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:39.374359 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:39.547254 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:39.547212 2560 generic.go:356] "Generic (PLEG): container finished" podID="34a812d1-d5c3-4506-9892-351debc35566" containerID="d13adb2c6174fad50c346f9032730d9779434851745e7608d54b583ae45bd533" exitCode=0 Jul 29 14:01:39.547398 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:39.547272 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" event={"ID":"34a812d1-d5c3-4506-9892-351debc35566","Type":"ContainerDied","Data":"d13adb2c6174fad50c346f9032730d9779434851745e7608d54b583ae45bd533"} Jul 29 14:01:41.029302 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:41.029268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:41.029728 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:41.029317 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:41.029728 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.029433 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:41.029728 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.029441 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:41.029728 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.029500 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret podName:6ebe845f-c3d6-4eed-a05e-88b608cc827f nodeName:}" failed. No retries permitted until 2026-07-29 14:02:13.029482129 +0000 UTC m=+66.230983285 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret") pod "global-pull-secret-syncer-2vm4f" (UID: "6ebe845f-c3d6-4eed-a05e-88b608cc827f") : object "kube-system"/"original-pull-secret" not registered Jul 29 14:01:41.029728 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.029518 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs podName:379c825d-878f-4066-841c-084c71477f94 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:13.029509069 +0000 UTC m=+66.231010225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs") pod "network-metrics-daemon-vfv5m" (UID: "379c825d-878f-4066-841c-084c71477f94") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 14:01:41.129662 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:41.129626 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:41.129840 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.129758 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 14:01:41.129840 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.129782 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 14:01:41.129840 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.129792 2560 projected.go:196] Error preparing data for projected volume kube-api-access-7kl8n for pod openshift-network-diagnostics/network-check-target-8qt8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:41.129977 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.129845 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n podName:7863615c-db3b-4db2-976d-22e559411059 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:13.129829074 +0000 UTC m=+66.331330225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7kl8n" (UniqueName: "kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n") pod "network-check-target-8qt8j" (UID: "7863615c-db3b-4db2-976d-22e559411059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 14:01:41.374981 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:41.374897 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:41.374981 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:41.374931 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:41.375231 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:41.374898 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:41.375231 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.375045 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vm4f" podUID="6ebe845f-c3d6-4eed-a05e-88b608cc827f" Jul 29 14:01:41.375231 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.375141 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfv5m" podUID="379c825d-878f-4066-841c-084c71477f94" Jul 29 14:01:41.375231 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:41.375221 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qt8j" podUID="7863615c-db3b-4db2-976d-22e559411059" Jul 29 14:01:42.077858 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.077830 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-155.ec2.internal" event="NodeReady" Jul 29 14:01:42.078519 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.078009 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 14:01:42.119329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.119289 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75f56884f8-77qwv"] Jul 29 14:01:42.140980 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.140957 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75f56884f8-77qwv"] Jul 29 14:01:42.141113 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.140988 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nm2x6"] Jul 29 14:01:42.141113 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.141068 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.144768 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.143708 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 14:01:42.144768 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.143833 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 14:01:42.144768 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.143862 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fnsd9\"" Jul 29 14:01:42.144768 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.144047 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 14:01:42.151663 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.151634 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 14:01:42.168244 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.168225 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sbgpj"] Jul 29 14:01:42.168352 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.168337 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nm2x6" Jul 29 14:01:42.171526 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.171508 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 14:01:42.171795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.171775 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 14:01:42.171918 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.171785 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vwvk9\"" Jul 29 14:01:42.177121 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.177101 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 14:01:42.186036 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.186020 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nm2x6"] Jul 29 14:01:42.186128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.186047 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sbgpj"] Jul 29 14:01:42.186128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.186124 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.188240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.188221 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v4vw8\"" Jul 29 14:01:42.188325 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.188281 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 14:01:42.208105 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.208078 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 14:01:42.237099 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237073 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-ca-trust-extracted\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.237220 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237106 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktjzz\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-kube-api-access-ktjzz\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.237220 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237185 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c6e3418-a206-431f-aad9-c917dd9df3f4-cert\") pod \"ingress-canary-nm2x6\" (UID: \"0c6e3418-a206-431f-aad9-c917dd9df3f4\") " pod="openshift-ingress-canary/ingress-canary-nm2x6" Jul 29 14:01:42.237220 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-image-registry-private-configuration\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.237379 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237239 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-certificates\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.237379 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237263 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-trusted-ca\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.237379 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237317 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.237379 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237371 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxkbj\" (UniqueName: \"kubernetes.io/projected/0c6e3418-a206-431f-aad9-c917dd9df3f4-kube-api-access-cxkbj\") pod \"ingress-canary-nm2x6\" (UID: \"0c6e3418-a206-431f-aad9-c917dd9df3f4\") " pod="openshift-ingress-canary/ingress-canary-nm2x6" Jul 29 14:01:42.237533 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237396 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-installation-pull-secrets\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.237533 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.237427 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-bound-sa-token\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338000 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.337922 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.338000 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.337963 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-ca-trust-extracted\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338000 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.337980 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktjzz\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-kube-api-access-ktjzz\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338068 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5rdq\" (UniqueName: \"kubernetes.io/projected/7010f78e-9440-452f-9616-8abac941737b-kube-api-access-v5rdq\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.338262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c6e3418-a206-431f-aad9-c917dd9df3f4-cert\") pod \"ingress-canary-nm2x6\" (UID: \"0c6e3418-a206-431f-aad9-c917dd9df3f4\") " pod="openshift-ingress-canary/ingress-canary-nm2x6" Jul 29 14:01:42.338262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-image-registry-private-configuration\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338258 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-certificates\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338475 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338285 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-trusted-ca\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338475 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338312 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7010f78e-9440-452f-9616-8abac941737b-config-volume\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.338475 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338349 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338475 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338373 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-ca-trust-extracted\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338475 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338375 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7010f78e-9440-452f-9616-8abac941737b-tmp-dir\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.338475 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxkbj\" (UniqueName: \"kubernetes.io/projected/0c6e3418-a206-431f-aad9-c917dd9df3f4-kube-api-access-cxkbj\") pod \"ingress-canary-nm2x6\" (UID: \"0c6e3418-a206-431f-aad9-c917dd9df3f4\") " pod="openshift-ingress-canary/ingress-canary-nm2x6" Jul 29 14:01:42.338475 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.338470 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 14:01:42.338840 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338478 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-installation-pull-secrets\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338840 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338511 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-bound-sa-token\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.338840 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.338484 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f56884f8-77qwv: secret "image-registry-tls" not found Jul 29 14:01:42.338991 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.338873 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls podName:07e74d63-8e6d-4ccf-85a9-33c75ba7baa0 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:42.83885301 +0000 UTC m=+36.040354188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls") pod "image-registry-75f56884f8-77qwv" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0") : secret "image-registry-tls" not found Jul 29 14:01:42.338991 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.338898 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-certificates\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.339231 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.339200 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-trusted-ca\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.342826 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.342804 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-image-registry-private-configuration\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.342987 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.342931 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c6e3418-a206-431f-aad9-c917dd9df3f4-cert\") pod \"ingress-canary-nm2x6\" (UID: \"0c6e3418-a206-431f-aad9-c917dd9df3f4\") " pod="openshift-ingress-canary/ingress-canary-nm2x6" Jul 29 14:01:42.343606 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.343586 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-installation-pull-secrets\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.347503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.347309 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktjzz\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-kube-api-access-ktjzz\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.347503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.347432 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-bound-sa-token\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.347503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.347478 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxkbj\" (UniqueName: \"kubernetes.io/projected/0c6e3418-a206-431f-aad9-c917dd9df3f4-kube-api-access-cxkbj\") pod \"ingress-canary-nm2x6\" (UID: \"0c6e3418-a206-431f-aad9-c917dd9df3f4\") " pod="openshift-ingress-canary/ingress-canary-nm2x6" Jul 29 14:01:42.439783 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.439754 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5rdq\" (UniqueName: \"kubernetes.io/projected/7010f78e-9440-452f-9616-8abac941737b-kube-api-access-v5rdq\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.439961 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.439800 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7010f78e-9440-452f-9616-8abac941737b-config-volume\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.439961 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.439844 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7010f78e-9440-452f-9616-8abac941737b-tmp-dir\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.439961 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.439895 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.440127 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.439984 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:42.440127 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.440036 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls podName:7010f78e-9440-452f-9616-8abac941737b nodeName:}" failed. No retries permitted until 2026-07-29 14:01:42.940019335 +0000 UTC m=+36.141520490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls") pod "dns-default-sbgpj" (UID: "7010f78e-9440-452f-9616-8abac941737b") : secret "dns-default-metrics-tls" not found Jul 29 14:01:42.440527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.440500 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7010f78e-9440-452f-9616-8abac941737b-tmp-dir\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.440774 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.440749 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7010f78e-9440-452f-9616-8abac941737b-config-volume\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.448407 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.448384 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5rdq\" (UniqueName: \"kubernetes.io/projected/7010f78e-9440-452f-9616-8abac941737b-kube-api-access-v5rdq\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.477471 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.477448 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nm2x6" Jul 29 14:01:42.685751 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.685585 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nm2x6"] Jul 29 14:01:42.688926 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:42.688899 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6e3418_a206_431f_aad9_c917dd9df3f4.slice/crio-16a31f8f43daec4b45a091a20b819f3b55dbc6782e63f625bfc4626bf6a45ed3 WatchSource:0}: Error finding container 16a31f8f43daec4b45a091a20b819f3b55dbc6782e63f625bfc4626bf6a45ed3: Status 404 returned error can't find the container with id 16a31f8f43daec4b45a091a20b819f3b55dbc6782e63f625bfc4626bf6a45ed3 Jul 29 14:01:42.841952 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.841915 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:42.842130 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.842082 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 14:01:42.842130 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.842104 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f56884f8-77qwv: secret "image-registry-tls" not found Jul 29 14:01:42.842220 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.842169 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls podName:07e74d63-8e6d-4ccf-85a9-33c75ba7baa0 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:43.842153086 +0000 UTC m=+37.043654239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls") pod "image-registry-75f56884f8-77qwv" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0") : secret "image-registry-tls" not found Jul 29 14:01:42.943139 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:42.943110 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:42.943321 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.943292 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:42.943373 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:42.943366 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls podName:7010f78e-9440-452f-9616-8abac941737b nodeName:}" failed. No retries permitted until 2026-07-29 14:01:43.943341879 +0000 UTC m=+37.144843036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls") pod "dns-default-sbgpj" (UID: "7010f78e-9440-452f-9616-8abac941737b") : secret "dns-default-metrics-tls" not found Jul 29 14:01:43.374073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.373990 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:01:43.374073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.374042 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:01:43.374674 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.373990 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:01:43.379594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.377989 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6wqn8\"" Jul 29 14:01:43.379594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.378245 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 14:01:43.379594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.378516 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7g4zm\"" Jul 29 14:01:43.379594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.378945 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 14:01:43.379594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.379274 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 14:01:43.379594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.379310 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 14:01:43.559739 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.559694 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nm2x6" event={"ID":"0c6e3418-a206-431f-aad9-c917dd9df3f4","Type":"ContainerStarted","Data":"16a31f8f43daec4b45a091a20b819f3b55dbc6782e63f625bfc4626bf6a45ed3"} Jul 29 14:01:43.850722 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.850687 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:43.850920 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:43.850809 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 14:01:43.850920 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:43.850821 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f56884f8-77qwv: secret "image-registry-tls" not found Jul 29 14:01:43.850920 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:43.850871 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls podName:07e74d63-8e6d-4ccf-85a9-33c75ba7baa0 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:45.850857224 +0000 UTC m=+39.052358375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls") pod "image-registry-75f56884f8-77qwv" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0") : secret "image-registry-tls" not found Jul 29 14:01:43.951569 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:43.951524 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:43.951731 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:43.951712 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:43.951823 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:43.951794 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls podName:7010f78e-9440-452f-9616-8abac941737b nodeName:}" failed. No retries permitted until 2026-07-29 14:01:45.951771634 +0000 UTC m=+39.153272807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls") pod "dns-default-sbgpj" (UID: "7010f78e-9440-452f-9616-8abac941737b") : secret "dns-default-metrics-tls" not found Jul 29 14:01:44.372572 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.372522 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv"] Jul 29 14:01:44.387047 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.387016 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n"] Jul 29 14:01:44.387494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.387272 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.389971 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.389792 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:01:44.389971 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.389792 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 29 14:01:44.389971 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.389890 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4q5l6\"" Jul 29 14:01:44.390307 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.389984 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 29 14:01:44.390307 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.390044 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 29 14:01:44.407240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.407219 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6"] Jul 29 14:01:44.407361 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.407328 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n" Jul 29 14:01:44.409940 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.409924 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 29 14:01:44.410054 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.410036 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:01:44.410101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.410043 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5pvs2\"" Jul 29 14:01:44.433487 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.433463 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv"] Jul 29 14:01:44.433596 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.433497 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n"] Jul 29 14:01:44.433596 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.433516 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6"] Jul 29 14:01:44.433596 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.433516 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:44.436920 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.436900 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 29 14:01:44.437018 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.436903 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 14:01:44.437074 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.436982 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gkpwm\"" Jul 29 14:01:44.437162 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.437147 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 29 14:01:44.437210 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.437188 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 14:01:44.455713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.455681 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrn8z\" (UniqueName: \"kubernetes.io/projected/7d5798e2-d0d9-4635-9dfa-f892e92bfc9f-kube-api-access-mrn8z\") pod \"service-ca-operator-5d6c75f84f-q9gzv\" (UID: \"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.455870 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.455722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d5798e2-d0d9-4635-9dfa-f892e92bfc9f-config\") pod \"service-ca-operator-5d6c75f84f-q9gzv\" (UID: \"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.455870 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.455793 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5798e2-d0d9-4635-9dfa-f892e92bfc9f-serving-cert\") pod \"service-ca-operator-5d6c75f84f-q9gzv\" (UID: \"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.466501 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.466474 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5"] Jul 29 14:01:44.486668 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.486643 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5" Jul 29 14:01:44.486821 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.486649 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-zn2dp"] Jul 29 14:01:44.489334 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.489310 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kdmxb\"" Jul 29 14:01:44.506998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.506973 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n"] Jul 29 14:01:44.507110 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.507085 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.509583 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.509515 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:01:44.509725 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.509585 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 29 14:01:44.509725 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.509596 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 29 14:01:44.509725 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.509587 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vv29l\"" Jul 29 14:01:44.509900 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.509763 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 29 14:01:44.517609 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.516899 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 29 14:01:44.525165 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.525143 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-78f77ccbf5-cjvmf"] Jul 29 14:01:44.525278 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.525256 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:44.527705 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.527684 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:01:44.527833 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.527722 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 29 14:01:44.527833 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.527684 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 29 14:01:44.527833 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.527775 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8plqs\"" Jul 29 14:01:44.546610 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.546577 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-pt4nq"] Jul 29 14:01:44.546765 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.546659 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.549558 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.549508 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 29 14:01:44.549678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.549605 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 29 14:01:44.549925 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.549907 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 29 14:01:44.549925 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.549921 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 14:01:44.550267 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.550072 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n5l9g\"" Jul 29 14:01:44.550267 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.550177 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 29 14:01:44.550492 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.550478 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 14:01:44.556101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.556074 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5798e2-d0d9-4635-9dfa-f892e92bfc9f-serving-cert\") pod \"service-ca-operator-5d6c75f84f-q9gzv\" (UID: \"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.556180 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.556147 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:44.556231 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.556179 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:44.556286 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.556226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrn8z\" (UniqueName: \"kubernetes.io/projected/7d5798e2-d0d9-4635-9dfa-f892e92bfc9f-kube-api-access-mrn8z\") pod \"service-ca-operator-5d6c75f84f-q9gzv\" (UID: \"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.556286 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.556252 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m982d\" (UniqueName: \"kubernetes.io/projected/ca7f7b51-a0e9-444f-8360-02e1976a84d6-kube-api-access-m982d\") pod \"volume-data-source-validator-549c6d54cf-p5n9n\" (UID: \"ca7f7b51-a0e9-444f-8360-02e1976a84d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n" Jul 29 14:01:44.556381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.556295 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d5798e2-d0d9-4635-9dfa-f892e92bfc9f-config\") pod \"service-ca-operator-5d6c75f84f-q9gzv\" (UID: \"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.556381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.556322 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzkzc\" (UniqueName: \"kubernetes.io/projected/5572d921-bddf-4317-b61e-fe21286426cd-kube-api-access-mzkzc\") pod \"network-check-source-7d88cb55d8-dl9r5\" (UID: \"5572d921-bddf-4317-b61e-fe21286426cd\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5" Jul 29 14:01:44.556481 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.556377 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqxbf\" (UniqueName: \"kubernetes.io/projected/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-kube-api-access-vqxbf\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:44.556778 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.556762 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d5798e2-d0d9-4635-9dfa-f892e92bfc9f-config\") pod \"service-ca-operator-5d6c75f84f-q9gzv\" (UID: \"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.559139 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.559113 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5798e2-d0d9-4635-9dfa-f892e92bfc9f-serving-cert\") pod \"service-ca-operator-5d6c75f84f-q9gzv\" (UID: \"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.570828 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.570800 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5"] Jul 29 14:01:44.570828 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.570830 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-zn2dp"] Jul 29 14:01:44.571159 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.570845 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n"] Jul 29 14:01:44.571159 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.570870 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78f77ccbf5-cjvmf"] Jul 29 14:01:44.571159 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.570884 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-pt4nq"] Jul 29 14:01:44.571159 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.570964 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.572862 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.572799 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrn8z\" (UniqueName: \"kubernetes.io/projected/7d5798e2-d0d9-4635-9dfa-f892e92bfc9f-kube-api-access-mrn8z\") pod \"service-ca-operator-5d6c75f84f-q9gzv\" (UID: \"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.573301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.573266 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 29 14:01:44.573671 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.573499 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 14:01:44.573671 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.573515 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 29 14:01:44.573671 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.573528 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 14:01:44.573985 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.573855 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hvgf4\"" Jul 29 14:01:44.579099 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.579081 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 29 14:01:44.657166 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657087 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-stats-auth\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.657166 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657141 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:44.657339 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657205 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzvh\" (UniqueName: \"kubernetes.io/projected/d7e9787c-6efa-4add-ab5f-81b01326487a-kube-api-access-7tzvh\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:44.657339 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657252 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58f1ae35-67f6-46a3-9a41-7cd808fe739c-serving-cert\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.657339 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.657339 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657294 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58f1ae35-67f6-46a3-9a41-7cd808fe739c-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.657339 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657314 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b63ed94d-a128-4c71-8465-479c86c3cc34-serving-cert\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.657479 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657346 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9fn\" (UniqueName: \"kubernetes.io/projected/e04e6603-8385-4ffb-aa85-ccae91e7658f-kube-api-access-8t9fn\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.657479 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657365 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58f1ae35-67f6-46a3-9a41-7cd808fe739c-service-ca-bundle\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.657479 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657394 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqxbf\" (UniqueName: \"kubernetes.io/projected/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-kube-api-access-vqxbf\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:44.657479 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-default-certificate\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.657616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657501 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b63ed94d-a128-4c71-8465-479c86c3cc34-trusted-ca\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.657616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657562 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:44.657616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657587 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b63ed94d-a128-4c71-8465-479c86c3cc34-config\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.657616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657606 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:44.657773 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657622 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c694m\" (UniqueName: \"kubernetes.io/projected/58f1ae35-67f6-46a3-9a41-7cd808fe739c-kube-api-access-c694m\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.657773 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657656 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjwck\" (UniqueName: \"kubernetes.io/projected/b63ed94d-a128-4c71-8465-479c86c3cc34-kube-api-access-tjwck\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.657773 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657675 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58f1ae35-67f6-46a3-9a41-7cd808fe739c-tmp\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.657773 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:44.657719 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:44.657917 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657722 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m982d\" (UniqueName: \"kubernetes.io/projected/ca7f7b51-a0e9-444f-8360-02e1976a84d6-kube-api-access-m982d\") pod \"volume-data-source-validator-549c6d54cf-p5n9n\" (UID: \"ca7f7b51-a0e9-444f-8360-02e1976a84d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n" Jul 29 14:01:44.657917 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:44.657789 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls podName:a48e2b71-ac38-43d0-8842-9c4e2c9cd833 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:45.157774579 +0000 UTC m=+38.359275728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xfxm6" (UID: "a48e2b71-ac38-43d0-8842-9c4e2c9cd833") : secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:44.657917 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657827 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzkzc\" (UniqueName: \"kubernetes.io/projected/5572d921-bddf-4317-b61e-fe21286426cd-kube-api-access-mzkzc\") pod \"network-check-source-7d88cb55d8-dl9r5\" (UID: \"5572d921-bddf-4317-b61e-fe21286426cd\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5" Jul 29 14:01:44.657917 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657848 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.657917 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.657864 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58f1ae35-67f6-46a3-9a41-7cd808fe739c-snapshots\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.658244 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.658227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:44.667019 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.666989 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqxbf\" (UniqueName: \"kubernetes.io/projected/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-kube-api-access-vqxbf\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:44.667019 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.667004 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzkzc\" (UniqueName: \"kubernetes.io/projected/5572d921-bddf-4317-b61e-fe21286426cd-kube-api-access-mzkzc\") pod \"network-check-source-7d88cb55d8-dl9r5\" (UID: \"5572d921-bddf-4317-b61e-fe21286426cd\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5" Jul 29 14:01:44.667185 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.667099 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m982d\" (UniqueName: \"kubernetes.io/projected/ca7f7b51-a0e9-444f-8360-02e1976a84d6-kube-api-access-m982d\") pod \"volume-data-source-validator-549c6d54cf-p5n9n\" (UID: \"ca7f7b51-a0e9-444f-8360-02e1976a84d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n" Jul 29 14:01:44.696794 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.696771 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" Jul 29 14:01:44.715511 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.715496 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n" Jul 29 14:01:44.758850 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.758801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-default-certificate\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.758977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.758864 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b63ed94d-a128-4c71-8465-479c86c3cc34-trusted-ca\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.758977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.758923 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b63ed94d-a128-4c71-8465-479c86c3cc34-config\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.758977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.758950 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c694m\" (UniqueName: \"kubernetes.io/projected/58f1ae35-67f6-46a3-9a41-7cd808fe739c-kube-api-access-c694m\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.759188 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.758996 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjwck\" (UniqueName: \"kubernetes.io/projected/b63ed94d-a128-4c71-8465-479c86c3cc34-kube-api-access-tjwck\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.759188 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759025 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58f1ae35-67f6-46a3-9a41-7cd808fe739c-tmp\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.759188 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759086 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.759188 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759110 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58f1ae35-67f6-46a3-9a41-7cd808fe739c-snapshots\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.759188 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759145 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-stats-auth\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.759188 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759185 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:44.759483 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759214 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzvh\" (UniqueName: \"kubernetes.io/projected/d7e9787c-6efa-4add-ab5f-81b01326487a-kube-api-access-7tzvh\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:44.759483 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759250 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58f1ae35-67f6-46a3-9a41-7cd808fe739c-serving-cert\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.759483 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759290 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.759483 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759314 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58f1ae35-67f6-46a3-9a41-7cd808fe739c-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.759483 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759345 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b63ed94d-a128-4c71-8465-479c86c3cc34-serving-cert\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.759483 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759373 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9fn\" (UniqueName: \"kubernetes.io/projected/e04e6603-8385-4ffb-aa85-ccae91e7658f-kube-api-access-8t9fn\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.759483 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759402 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58f1ae35-67f6-46a3-9a41-7cd808fe739c-service-ca-bundle\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.759840 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.759813 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58f1ae35-67f6-46a3-9a41-7cd808fe739c-tmp\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.760027 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:44.760008 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 14:01:44.760114 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:44.760071 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 14:01:44.760166 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:44.760074 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls podName:d7e9787c-6efa-4add-ab5f-81b01326487a nodeName:}" failed. No retries permitted until 2026-07-29 14:01:45.260055808 +0000 UTC m=+38.461556960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls") pod "cluster-samples-operator-6748489656-dbx8n" (UID: "d7e9787c-6efa-4add-ab5f-81b01326487a") : secret "samples-operator-tls" not found Jul 29 14:01:44.760166 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:44.760151 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:45.260135782 +0000 UTC m=+38.461636938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : secret "router-metrics-certs-default" not found Jul 29 14:01:44.760324 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.760300 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58f1ae35-67f6-46a3-9a41-7cd808fe739c-service-ca-bundle\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.760454 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:44.760433 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:45.260417139 +0000 UTC m=+38.461918290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : configmap references non-existent config key: service-ca.crt Jul 29 14:01:44.760645 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.760005 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58f1ae35-67f6-46a3-9a41-7cd808fe739c-snapshots\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.761269 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.761239 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b63ed94d-a128-4c71-8465-479c86c3cc34-config\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.761471 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.761446 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b63ed94d-a128-4c71-8465-479c86c3cc34-trusted-ca\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.761655 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.761631 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58f1ae35-67f6-46a3-9a41-7cd808fe739c-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.762501 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.762479 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-default-certificate\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.763339 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.763313 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58f1ae35-67f6-46a3-9a41-7cd808fe739c-serving-cert\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.763801 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.763761 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b63ed94d-a128-4c71-8465-479c86c3cc34-serving-cert\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.764831 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.764785 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-stats-auth\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.770015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.769601 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzvh\" (UniqueName: \"kubernetes.io/projected/d7e9787c-6efa-4add-ab5f-81b01326487a-kube-api-access-7tzvh\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:44.770015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.769981 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjwck\" (UniqueName: \"kubernetes.io/projected/b63ed94d-a128-4c71-8465-479c86c3cc34-kube-api-access-tjwck\") pod \"console-operator-584dcb6b54-zn2dp\" (UID: \"b63ed94d-a128-4c71-8465-479c86c3cc34\") " pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.770636 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.770618 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9fn\" (UniqueName: \"kubernetes.io/projected/e04e6603-8385-4ffb-aa85-ccae91e7658f-kube-api-access-8t9fn\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:44.770767 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.770727 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c694m\" (UniqueName: \"kubernetes.io/projected/58f1ae35-67f6-46a3-9a41-7cd808fe739c-kube-api-access-c694m\") pod \"insights-operator-5885fdb759-pt4nq\" (UID: \"58f1ae35-67f6-46a3-9a41-7cd808fe739c\") " pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.799695 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.799653 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5" Jul 29 14:01:44.819588 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.819496 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:44.876339 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.876273 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv"] Jul 29 14:01:44.884629 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.884600 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-pt4nq" Jul 29 14:01:44.902796 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.902674 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n"] Jul 29 14:01:44.988633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:44.985401 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5"] Jul 29 14:01:45.013790 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:45.013711 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7f7b51_a0e9_444f_8360_02e1976a84d6.slice/crio-33e7269ee97e5a96bc4ea437edd81344ec9b5a5cc3e48b1bf39daf05f12ac275 WatchSource:0}: Error finding container 33e7269ee97e5a96bc4ea437edd81344ec9b5a5cc3e48b1bf39daf05f12ac275: Status 404 returned error can't find the container with id 33e7269ee97e5a96bc4ea437edd81344ec9b5a5cc3e48b1bf39daf05f12ac275 Jul 29 14:01:45.013974 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:45.013896 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5798e2_d0d9_4635_9dfa_f892e92bfc9f.slice/crio-388b28d62ec1c113f7dfe974ea706efc1b50e3c19370655e044c608e319780ab WatchSource:0}: Error finding container 388b28d62ec1c113f7dfe974ea706efc1b50e3c19370655e044c608e319780ab: Status 404 returned error can't find the container with id 388b28d62ec1c113f7dfe974ea706efc1b50e3c19370655e044c608e319780ab Jul 29 14:01:45.014207 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:45.014188 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5572d921_bddf_4317_b61e_fe21286426cd.slice/crio-4e3341f6af41b7885b114fc507c0f08d7a89890c603e67c7bdabfab44b5a3dba WatchSource:0}: Error finding container 4e3341f6af41b7885b114fc507c0f08d7a89890c603e67c7bdabfab44b5a3dba: Status 404 returned error can't find the container with id 4e3341f6af41b7885b114fc507c0f08d7a89890c603e67c7bdabfab44b5a3dba Jul 29 14:01:45.120149 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.119908 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-pt4nq"] Jul 29 14:01:45.128029 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.128003 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-zn2dp"] Jul 29 14:01:45.132046 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:01:45.132019 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f1ae35_67f6_46a3_9a41_7cd808fe739c.slice/crio-9a42050d6bdefc77984b07dbbe9c5f181795778393270fadaf701dc3ff068bb8 WatchSource:0}: Error finding container 9a42050d6bdefc77984b07dbbe9c5f181795778393270fadaf701dc3ff068bb8: Status 404 returned error can't find the container with id 9a42050d6bdefc77984b07dbbe9c5f181795778393270fadaf701dc3ff068bb8 Jul 29 14:01:45.163175 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.163109 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:45.163389 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.163274 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:45.163389 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.163345 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls podName:a48e2b71-ac38-43d0-8842-9c4e2c9cd833 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:46.163328554 +0000 UTC m=+39.364829705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xfxm6" (UID: "a48e2b71-ac38-43d0-8842-9c4e2c9cd833") : secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:45.264591 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.264531 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:45.264772 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.264627 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:45.264772 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.264698 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:45.264772 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.264701 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 14:01:45.264772 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.264744 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 14:01:45.264932 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.264792 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:46.264778187 +0000 UTC m=+39.466279337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : secret "router-metrics-certs-default" not found Jul 29 14:01:45.264932 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.264805 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls podName:d7e9787c-6efa-4add-ab5f-81b01326487a nodeName:}" failed. No retries permitted until 2026-07-29 14:01:46.264799031 +0000 UTC m=+39.466300181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls") pod "cluster-samples-operator-6748489656-dbx8n" (UID: "d7e9787c-6efa-4add-ab5f-81b01326487a") : secret "samples-operator-tls" not found Jul 29 14:01:45.264932 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.264833 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:46.264812715 +0000 UTC m=+39.466313871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : configmap references non-existent config key: service-ca.crt Jul 29 14:01:45.568055 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.568023 2560 generic.go:356] "Generic (PLEG): container finished" podID="34a812d1-d5c3-4506-9892-351debc35566" containerID="52d0e7ff59019463c54af67a894127d85c8604b09e48fc1db7de7eb1a0785bf2" exitCode=0 Jul 29 14:01:45.568502 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.568102 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" event={"ID":"34a812d1-d5c3-4506-9892-351debc35566","Type":"ContainerDied","Data":"52d0e7ff59019463c54af67a894127d85c8604b09e48fc1db7de7eb1a0785bf2"} Jul 29 14:01:45.569319 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.569295 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-pt4nq" event={"ID":"58f1ae35-67f6-46a3-9a41-7cd808fe739c","Type":"ContainerStarted","Data":"9a42050d6bdefc77984b07dbbe9c5f181795778393270fadaf701dc3ff068bb8"} Jul 29 14:01:45.570488 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.570459 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" event={"ID":"b63ed94d-a128-4c71-8465-479c86c3cc34","Type":"ContainerStarted","Data":"75588c21f1f19f1481d15a64fa235a3cf4cdba85905bbb079b7997875047847b"} Jul 29 14:01:45.571843 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.571822 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5" event={"ID":"5572d921-bddf-4317-b61e-fe21286426cd","Type":"ContainerStarted","Data":"4e3341f6af41b7885b114fc507c0f08d7a89890c603e67c7bdabfab44b5a3dba"} Jul 29 14:01:45.573098 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.573061 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" event={"ID":"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f","Type":"ContainerStarted","Data":"388b28d62ec1c113f7dfe974ea706efc1b50e3c19370655e044c608e319780ab"} Jul 29 14:01:45.574214 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.574194 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n" event={"ID":"ca7f7b51-a0e9-444f-8360-02e1976a84d6","Type":"ContainerStarted","Data":"33e7269ee97e5a96bc4ea437edd81344ec9b5a5cc3e48b1bf39daf05f12ac275"} Jul 29 14:01:45.870392 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.870296 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:45.870569 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.870495 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 14:01:45.870569 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.870514 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f56884f8-77qwv: secret "image-registry-tls" not found Jul 29 14:01:45.870699 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.870595 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls podName:07e74d63-8e6d-4ccf-85a9-33c75ba7baa0 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:49.87057613 +0000 UTC m=+43.072077286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls") pod "image-registry-75f56884f8-77qwv" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0") : secret "image-registry-tls" not found Jul 29 14:01:45.971330 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:45.971293 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:45.971575 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.971512 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:45.971654 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:45.971595 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls podName:7010f78e-9440-452f-9616-8abac941737b nodeName:}" failed. No retries permitted until 2026-07-29 14:01:49.97157429 +0000 UTC m=+43.173075445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls") pod "dns-default-sbgpj" (UID: "7010f78e-9440-452f-9616-8abac941737b") : secret "dns-default-metrics-tls" not found Jul 29 14:01:46.173752 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:46.173656 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:46.173894 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:46.173827 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:46.173946 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:46.173900 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls podName:a48e2b71-ac38-43d0-8842-9c4e2c9cd833 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:48.173880755 +0000 UTC m=+41.375381908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xfxm6" (UID: "a48e2b71-ac38-43d0-8842-9c4e2c9cd833") : secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:46.274936 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:46.274490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:46.274936 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:46.274580 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:46.274936 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:46.274632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:46.274936 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:46.274652 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 14:01:46.274936 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:46.274729 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:48.27470932 +0000 UTC m=+41.476210487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : secret "router-metrics-certs-default" not found Jul 29 14:01:46.274936 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:46.274803 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:48.274787143 +0000 UTC m=+41.476288295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : configmap references non-existent config key: service-ca.crt Jul 29 14:01:46.274936 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:46.274874 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 14:01:46.274936 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:46.274911 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls podName:d7e9787c-6efa-4add-ab5f-81b01326487a nodeName:}" failed. No retries permitted until 2026-07-29 14:01:48.274897883 +0000 UTC m=+41.476399040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls") pod "cluster-samples-operator-6748489656-dbx8n" (UID: "d7e9787c-6efa-4add-ab5f-81b01326487a") : secret "samples-operator-tls" not found Jul 29 14:01:46.583569 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:46.582648 2560 generic.go:356] "Generic (PLEG): container finished" podID="34a812d1-d5c3-4506-9892-351debc35566" containerID="8c65e2e1537ab67073bdcc02d7b4b9d30547d3ae52710671b7fecbff9505882e" exitCode=0 Jul 29 14:01:46.583569 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:46.582698 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" event={"ID":"34a812d1-d5c3-4506-9892-351debc35566","Type":"ContainerDied","Data":"8c65e2e1537ab67073bdcc02d7b4b9d30547d3ae52710671b7fecbff9505882e"} Jul 29 14:01:47.589698 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:47.589219 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n" event={"ID":"ca7f7b51-a0e9-444f-8360-02e1976a84d6","Type":"ContainerStarted","Data":"7ddcf125e9b33f5bce83d945a4995a951624d7a9ba95ee95aabb5c953b536edd"} Jul 29 14:01:47.593312 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:47.593277 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" event={"ID":"34a812d1-d5c3-4506-9892-351debc35566","Type":"ContainerStarted","Data":"edb5819732c7a23559d19b8df2377d7a458caa02b903475e965bbb144ccc631c"} Jul 29 14:01:47.605233 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:47.605178 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p5n9n" podStartSLOduration=1.535842326 podStartE2EDuration="3.605155974s" totalImagesPullingTime="2.069313648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:47.605124489 +0000 UTC m=+40.806625671" watchObservedRunningTime="2026-07-29 14:01:47.605155974 +0000 UTC m=+40.806657126" Jul 29 14:01:47.627210 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:47.627071 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rrvvf" podStartSLOduration=9.863843655 podStartE2EDuration="40.627051197s" totalImagesPullingTime="30.763207542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:47.625674133 +0000 UTC m=+40.827175320" watchObservedRunningTime="2026-07-29 14:01:47.627051197 +0000 UTC m=+40.828552370" Jul 29 14:01:48.193880 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:48.193844 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:48.194062 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:48.194036 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:48.194134 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:48.194121 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls podName:a48e2b71-ac38-43d0-8842-9c4e2c9cd833 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:52.194098557 +0000 UTC m=+45.395599711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xfxm6" (UID: "a48e2b71-ac38-43d0-8842-9c4e2c9cd833") : secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:48.295069 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:48.295023 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:48.295235 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:48.295100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:48.295235 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:48.295143 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:48.295235 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:48.295189 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 14:01:48.295235 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:48.295219 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 14:01:48.295421 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:48.295263 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:52.295243741 +0000 UTC m=+45.496744900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : secret "router-metrics-certs-default" not found Jul 29 14:01:48.295421 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:48.295281 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls podName:d7e9787c-6efa-4add-ab5f-81b01326487a nodeName:}" failed. No retries permitted until 2026-07-29 14:01:52.295274596 +0000 UTC m=+45.496775746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls") pod "cluster-samples-operator-6748489656-dbx8n" (UID: "d7e9787c-6efa-4add-ab5f-81b01326487a") : secret "samples-operator-tls" not found Jul 29 14:01:48.295421 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:48.295339 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:01:52.295321174 +0000 UTC m=+45.496822327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : configmap references non-existent config key: service-ca.crt Jul 29 14:01:49.908928 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:49.908673 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:49.909365 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:49.908826 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 14:01:49.909365 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:49.909027 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f56884f8-77qwv: secret "image-registry-tls" not found Jul 29 14:01:49.909365 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:49.909115 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls podName:07e74d63-8e6d-4ccf-85a9-33c75ba7baa0 nodeName:}" failed. No retries permitted until 2026-07-29 14:01:57.909091041 +0000 UTC m=+51.110592211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls") pod "image-registry-75f56884f8-77qwv" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0") : secret "image-registry-tls" not found Jul 29 14:01:50.010630 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:50.010592 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:50.010788 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:50.010752 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:50.010876 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:50.010837 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls podName:7010f78e-9440-452f-9616-8abac941737b nodeName:}" failed. No retries permitted until 2026-07-29 14:01:58.010816762 +0000 UTC m=+51.212317928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls") pod "dns-default-sbgpj" (UID: "7010f78e-9440-452f-9616-8abac941737b") : secret "dns-default-metrics-tls" not found Jul 29 14:01:52.233292 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.233141 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:01:52.233754 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:52.233327 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:52.233754 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:52.233412 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls podName:a48e2b71-ac38-43d0-8842-9c4e2c9cd833 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:00.233387464 +0000 UTC m=+53.434888626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xfxm6" (UID: "a48e2b71-ac38-43d0-8842-9c4e2c9cd833") : secret "cluster-monitoring-operator-tls" not found Jul 29 14:01:52.333852 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.333817 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:52.334015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.333880 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:01:52.334015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.333922 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:01:52.334015 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:52.333991 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 14:01:52.334147 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:52.334052 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 14:01:52.334147 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:52.334067 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:02:00.334046183 +0000 UTC m=+53.535547340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : secret "router-metrics-certs-default" not found Jul 29 14:01:52.334147 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:52.334085 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:02:00.334076836 +0000 UTC m=+53.535577986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : configmap references non-existent config key: service-ca.crt Jul 29 14:01:52.334147 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:52.334110 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls podName:d7e9787c-6efa-4add-ab5f-81b01326487a nodeName:}" failed. No retries permitted until 2026-07-29 14:02:00.33409362 +0000 UTC m=+53.535594775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls") pod "cluster-samples-operator-6748489656-dbx8n" (UID: "d7e9787c-6efa-4add-ab5f-81b01326487a") : secret "samples-operator-tls" not found Jul 29 14:01:52.609924 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.609874 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nm2x6" event={"ID":"0c6e3418-a206-431f-aad9-c917dd9df3f4","Type":"ContainerStarted","Data":"45cf52e3fe5a39107ecf0ed6147d076c5fa5049a86e11f12dc08e0e649ff9f13"} Jul 29 14:01:52.611792 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.611753 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-pt4nq" event={"ID":"58f1ae35-67f6-46a3-9a41-7cd808fe739c","Type":"ContainerStarted","Data":"5a5b0aa6f0ee512c14e37fb07746c2b37162119a0b586decb01f40f619bb315d"} Jul 29 14:01:52.613459 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.613433 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/0.log" Jul 29 14:01:52.613593 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.613469 2560 generic.go:356] "Generic (PLEG): container finished" podID="b63ed94d-a128-4c71-8465-479c86c3cc34" containerID="1c5deb636e27344cee83d370415c8789aa97600d1a72853c1e161aa62396faa2" exitCode=255 Jul 29 14:01:52.613802 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.613744 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" event={"ID":"b63ed94d-a128-4c71-8465-479c86c3cc34","Type":"ContainerDied","Data":"1c5deb636e27344cee83d370415c8789aa97600d1a72853c1e161aa62396faa2"} Jul 29 14:01:52.613802 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.613755 2560 scope.go:117] "RemoveContainer" containerID="1c5deb636e27344cee83d370415c8789aa97600d1a72853c1e161aa62396faa2" Jul 29 14:01:52.615062 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.615043 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5" event={"ID":"5572d921-bddf-4317-b61e-fe21286426cd","Type":"ContainerStarted","Data":"da72018331c536992385b85efde43cf6ed9b7762366cfd91b7032985571edbdb"} Jul 29 14:01:52.616585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.616566 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" event={"ID":"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f","Type":"ContainerStarted","Data":"7a0e210d3121454188e5af7edd5bcca59cc0f9ad272793bef3e72240fdb7b34b"} Jul 29 14:01:52.627227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.627179 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nm2x6" podStartSLOduration=1.43039081 podStartE2EDuration="10.627160677s" totalImagesPullingTime="9.196769867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:52.625648819 +0000 UTC m=+45.827149992" watchObservedRunningTime="2026-07-29 14:01:52.627160677 +0000 UTC m=+45.828661850" Jul 29 14:01:52.642609 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.642045 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" podStartSLOduration=1.7709456449999998 podStartE2EDuration="8.642023942s" totalImagesPullingTime="6.871078297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:52.640716693 +0000 UTC m=+45.842217868" watchObservedRunningTime="2026-07-29 14:01:52.642023942 +0000 UTC m=+45.843525094" Jul 29 14:01:52.660292 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.660245 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dl9r5" podStartSLOduration=1.7893198369999999 podStartE2EDuration="8.660228375s" totalImagesPullingTime="6.870908538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:52.659678416 +0000 UTC m=+45.861179590" watchObservedRunningTime="2026-07-29 14:01:52.660228375 +0000 UTC m=+45.861729547" Jul 29 14:01:52.693827 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:52.693717 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-pt4nq" podStartSLOduration=1.9345618340000001 podStartE2EDuration="8.693697148s" totalImagesPullingTime="6.759135314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:01:52.693656111 +0000 UTC m=+45.895157282" watchObservedRunningTime="2026-07-29 14:01:52.693697148 +0000 UTC m=+45.895198321" Jul 29 14:01:53.621600 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:53.621570 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/1.log" Jul 29 14:01:53.622124 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:53.621984 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/0.log" Jul 29 14:01:53.622124 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:53.622017 2560 generic.go:356] "Generic (PLEG): container finished" podID="b63ed94d-a128-4c71-8465-479c86c3cc34" containerID="1be502d6f9866e0efd89b17a045c3cfe71579600216081cc1ed728b46256f4b2" exitCode=255 Jul 29 14:01:53.622227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:53.622116 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" event={"ID":"b63ed94d-a128-4c71-8465-479c86c3cc34","Type":"ContainerDied","Data":"1be502d6f9866e0efd89b17a045c3cfe71579600216081cc1ed728b46256f4b2"} Jul 29 14:01:53.622227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:53.622170 2560 scope.go:117] "RemoveContainer" containerID="1c5deb636e27344cee83d370415c8789aa97600d1a72853c1e161aa62396faa2" Jul 29 14:01:53.622375 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:53.622357 2560 scope.go:117] "RemoveContainer" containerID="1be502d6f9866e0efd89b17a045c3cfe71579600216081cc1ed728b46256f4b2" Jul 29 14:01:53.622677 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:53.622648 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-zn2dp_openshift-console-operator(b63ed94d-a128-4c71-8465-479c86c3cc34)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" podUID="b63ed94d-a128-4c71-8465-479c86c3cc34" Jul 29 14:01:54.627757 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.627611 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/1.log" Jul 29 14:01:54.628163 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.628143 2560 scope.go:117] "RemoveContainer" containerID="1be502d6f9866e0efd89b17a045c3cfe71579600216081cc1ed728b46256f4b2" Jul 29 14:01:54.628350 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:54.628327 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-zn2dp_openshift-console-operator(b63ed94d-a128-4c71-8465-479c86c3cc34)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" podUID="b63ed94d-a128-4c71-8465-479c86c3cc34" Jul 29 14:01:54.707434 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.707398 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5b4fk"] Jul 29 14:01:54.730038 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.730001 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5b4fk"] Jul 29 14:01:54.730230 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.730179 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.733215 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.733189 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 14:01:54.733367 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.733291 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jrmfz\"" Jul 29 14:01:54.733451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.733434 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 14:01:54.820567 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.820517 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:54.820740 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.820617 2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:01:54.858458 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.858414 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q5wp\" (UniqueName: \"kubernetes.io/projected/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-api-access-6q5wp\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.858646 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.858468 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-data-volume\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.858646 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.858630 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-crio-socket\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.858724 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.858673 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.858758 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.858743 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.959964 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.959928 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-crio-socket\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.960168 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.959984 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.960168 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:54.960061 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 14:01:54.960168 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.960092 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.960168 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:54.960122 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm podName:1d771142-7766-4ff7-8a28-4b3eea8bdfbd nodeName:}" failed. No retries permitted until 2026-07-29 14:01:55.46010388 +0000 UTC m=+48.661605033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5b4fk" (UID: "1d771142-7766-4ff7-8a28-4b3eea8bdfbd") : configmap "kube-rbac-proxy" not found Jul 29 14:01:54.960168 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.960161 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q5wp\" (UniqueName: \"kubernetes.io/projected/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-api-access-6q5wp\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.960377 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.960185 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-data-volume\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.960377 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:54.960191 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:01:54.960377 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:54.960254 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls podName:1d771142-7766-4ff7-8a28-4b3eea8bdfbd nodeName:}" failed. No retries permitted until 2026-07-29 14:01:55.460239223 +0000 UTC m=+48.661740390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5b4fk" (UID: "1d771142-7766-4ff7-8a28-4b3eea8bdfbd") : secret "insights-runtime-extractor-tls" not found Jul 29 14:01:54.960377 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.960294 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-crio-socket\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.960578 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.960561 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-data-volume\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:54.977508 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:54.977473 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q5wp\" (UniqueName: \"kubernetes.io/projected/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-api-access-6q5wp\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:55.464516 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:55.464475 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:55.464703 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:55.464588 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:55.464703 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:55.464640 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 14:01:55.464793 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:55.464733 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm podName:1d771142-7766-4ff7-8a28-4b3eea8bdfbd nodeName:}" failed. No retries permitted until 2026-07-29 14:01:56.464712207 +0000 UTC m=+49.666213374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5b4fk" (UID: "1d771142-7766-4ff7-8a28-4b3eea8bdfbd") : configmap "kube-rbac-proxy" not found Jul 29 14:01:55.464793 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:55.464733 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:01:55.464793 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:55.464775 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls podName:1d771142-7766-4ff7-8a28-4b3eea8bdfbd nodeName:}" failed. No retries permitted until 2026-07-29 14:01:56.464765166 +0000 UTC m=+49.666266320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5b4fk" (UID: "1d771142-7766-4ff7-8a28-4b3eea8bdfbd") : secret "insights-runtime-extractor-tls" not found Jul 29 14:01:55.631456 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:55.631433 2560 scope.go:117] "RemoveContainer" containerID="1be502d6f9866e0efd89b17a045c3cfe71579600216081cc1ed728b46256f4b2" Jul 29 14:01:55.631846 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:55.631607 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-zn2dp_openshift-console-operator(b63ed94d-a128-4c71-8465-479c86c3cc34)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" podUID="b63ed94d-a128-4c71-8465-479c86c3cc34" Jul 29 14:01:56.474455 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:56.474422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:56.474679 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:56.474496 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:56.474679 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:56.474570 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 14:01:56.474679 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:56.474650 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm podName:1d771142-7766-4ff7-8a28-4b3eea8bdfbd nodeName:}" failed. No retries permitted until 2026-07-29 14:01:58.474630482 +0000 UTC m=+51.676131648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5b4fk" (UID: "1d771142-7766-4ff7-8a28-4b3eea8bdfbd") : configmap "kube-rbac-proxy" not found Jul 29 14:01:56.474679 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:56.474655 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:01:56.474897 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:56.474773 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls podName:1d771142-7766-4ff7-8a28-4b3eea8bdfbd nodeName:}" failed. No retries permitted until 2026-07-29 14:01:58.474718432 +0000 UTC m=+51.676219585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5b4fk" (UID: "1d771142-7766-4ff7-8a28-4b3eea8bdfbd") : secret "insights-runtime-extractor-tls" not found Jul 29 14:01:56.696884 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:56.696852 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w5jp9_19853864-4d88-4abe-8485-aa693e06ab08/dns-node-resolver/0.log" Jul 29 14:01:57.697709 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:57.697686 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v5hwn_e9a1dbd3-0eba-4141-a243-bfb4b7da0740/node-ca/0.log" Jul 29 14:01:57.987794 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:57.987713 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:01:57.987942 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:57.987899 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 14:01:57.987942 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:57.987919 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f56884f8-77qwv: secret "image-registry-tls" not found Jul 29 14:01:57.988011 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:57.987974 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls podName:07e74d63-8e6d-4ccf-85a9-33c75ba7baa0 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:13.987958492 +0000 UTC m=+67.189459641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls") pod "image-registry-75f56884f8-77qwv" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0") : secret "image-registry-tls" not found Jul 29 14:01:58.088468 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:58.088433 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:01:58.088634 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:58.088610 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 14:01:58.088681 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:58.088672 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls podName:7010f78e-9440-452f-9616-8abac941737b nodeName:}" failed. No retries permitted until 2026-07-29 14:02:14.088658507 +0000 UTC m=+67.290159659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls") pod "dns-default-sbgpj" (UID: "7010f78e-9440-452f-9616-8abac941737b") : secret "dns-default-metrics-tls" not found Jul 29 14:01:58.492716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:58.492683 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:58.492889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:58.492751 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:01:58.492889 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:58.492806 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 14:01:58.492889 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:58.492854 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:01:58.492889 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:58.492873 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm podName:1d771142-7766-4ff7-8a28-4b3eea8bdfbd nodeName:}" failed. No retries permitted until 2026-07-29 14:02:02.492856903 +0000 UTC m=+55.694358053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5b4fk" (UID: "1d771142-7766-4ff7-8a28-4b3eea8bdfbd") : configmap "kube-rbac-proxy" not found Jul 29 14:01:58.492889 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:01:58.492892 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls podName:1d771142-7766-4ff7-8a28-4b3eea8bdfbd nodeName:}" failed. No retries permitted until 2026-07-29 14:02:02.492881684 +0000 UTC m=+55.694382834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5b4fk" (UID: "1d771142-7766-4ff7-8a28-4b3eea8bdfbd") : secret "insights-runtime-extractor-tls" not found Jul 29 14:01:58.697798 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:01:58.697771 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nm2x6_0c6e3418-a206-431f-aad9-c917dd9df3f4/serve-healthcheck-canary/0.log" Jul 29 14:02:00.307557 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:00.307504 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:02:00.308089 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:00.307677 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 14:02:00.308089 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:00.307764 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls podName:a48e2b71-ac38-43d0-8842-9c4e2c9cd833 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:16.30774229 +0000 UTC m=+69.509243460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xfxm6" (UID: "a48e2b71-ac38-43d0-8842-9c4e2c9cd833") : secret "cluster-monitoring-operator-tls" not found Jul 29 14:02:00.408909 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:00.408873 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:00.409065 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:00.408931 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:02:00.409065 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:00.408957 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:00.409065 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:00.409030 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 14:02:00.409173 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:00.409074 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 14:02:00.409173 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:00.409076 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:02:16.40906306 +0000 UTC m=+69.610564210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : configmap references non-existent config key: service-ca.crt Jul 29 14:02:00.409173 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:00.409126 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs podName:e04e6603-8385-4ffb-aa85-ccae91e7658f nodeName:}" failed. No retries permitted until 2026-07-29 14:02:16.409113842 +0000 UTC m=+69.610614991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs") pod "router-default-78f77ccbf5-cjvmf" (UID: "e04e6603-8385-4ffb-aa85-ccae91e7658f") : secret "router-metrics-certs-default" not found Jul 29 14:02:00.409173 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:00.409137 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls podName:d7e9787c-6efa-4add-ab5f-81b01326487a nodeName:}" failed. No retries permitted until 2026-07-29 14:02:16.409130894 +0000 UTC m=+69.610632044 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls") pod "cluster-samples-operator-6748489656-dbx8n" (UID: "d7e9787c-6efa-4add-ab5f-81b01326487a") : secret "samples-operator-tls" not found Jul 29 14:02:02.529204 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:02.529169 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:02:02.529718 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:02.529283 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:02:02.529718 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:02.529342 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 14:02:02.529718 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:02.529427 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls podName:1d771142-7766-4ff7-8a28-4b3eea8bdfbd nodeName:}" failed. No retries permitted until 2026-07-29 14:02:10.529407501 +0000 UTC m=+63.730908960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5b4fk" (UID: "1d771142-7766-4ff7-8a28-4b3eea8bdfbd") : secret "insights-runtime-extractor-tls" not found Jul 29 14:02:02.529872 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:02.529837 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:02:07.548865 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:07.548837 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-plvns" Jul 29 14:02:10.374518 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.374486 2560 scope.go:117] "RemoveContainer" containerID="1be502d6f9866e0efd89b17a045c3cfe71579600216081cc1ed728b46256f4b2" Jul 29 14:02:10.601493 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.601455 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:02:10.603860 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.603843 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d771142-7766-4ff7-8a28-4b3eea8bdfbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5b4fk\" (UID: \"1d771142-7766-4ff7-8a28-4b3eea8bdfbd\") " pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:02:10.641890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.641816 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jrmfz\"" Jul 29 14:02:10.649702 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.649685 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5b4fk" Jul 29 14:02:10.677790 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.677763 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:02:10.678213 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.678198 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/1.log" Jul 29 14:02:10.678284 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.678230 2560 generic.go:356] "Generic (PLEG): container finished" podID="b63ed94d-a128-4c71-8465-479c86c3cc34" containerID="9f24da733116b8cb8be85b2f4808d743cbcb1c2b1faa1737323c15aa48f4c83f" exitCode=255 Jul 29 14:02:10.678320 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.678297 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" event={"ID":"b63ed94d-a128-4c71-8465-479c86c3cc34","Type":"ContainerDied","Data":"9f24da733116b8cb8be85b2f4808d743cbcb1c2b1faa1737323c15aa48f4c83f"} Jul 29 14:02:10.678360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.678349 2560 scope.go:117] "RemoveContainer" containerID="1be502d6f9866e0efd89b17a045c3cfe71579600216081cc1ed728b46256f4b2" Jul 29 14:02:10.678755 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.678736 2560 scope.go:117] "RemoveContainer" containerID="9f24da733116b8cb8be85b2f4808d743cbcb1c2b1faa1737323c15aa48f4c83f" Jul 29 14:02:10.678966 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:10.678944 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-zn2dp_openshift-console-operator(b63ed94d-a128-4c71-8465-479c86c3cc34)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" podUID="b63ed94d-a128-4c71-8465-479c86c3cc34" Jul 29 14:02:10.775366 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:10.775231 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5b4fk"] Jul 29 14:02:10.777851 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:10.777826 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d771142_7766_4ff7_8a28_4b3eea8bdfbd.slice/crio-58ef8fe5730b46c7e5e0d8e6616943d536021e8438de66d91d00e5e70a4b47b2 WatchSource:0}: Error finding container 58ef8fe5730b46c7e5e0d8e6616943d536021e8438de66d91d00e5e70a4b47b2: Status 404 returned error can't find the container with id 58ef8fe5730b46c7e5e0d8e6616943d536021e8438de66d91d00e5e70a4b47b2 Jul 29 14:02:11.683756 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:11.683611 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:02:11.685293 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:11.685244 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5b4fk" event={"ID":"1d771142-7766-4ff7-8a28-4b3eea8bdfbd","Type":"ContainerStarted","Data":"58ef8fe5730b46c7e5e0d8e6616943d536021e8438de66d91d00e5e70a4b47b2"} Jul 29 14:02:13.126660 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.126615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:02:13.127112 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.126715 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:02:13.130065 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.130029 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 14:02:13.130556 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.130510 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 14:02:13.139590 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.139562 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/379c825d-878f-4066-841c-084c71477f94-metrics-certs\") pod \"network-metrics-daemon-vfv5m\" (UID: \"379c825d-878f-4066-841c-084c71477f94\") " pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:02:13.139733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.139705 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ebe845f-c3d6-4eed-a05e-88b608cc827f-original-pull-secret\") pod \"global-pull-secret-syncer-2vm4f\" (UID: \"6ebe845f-c3d6-4eed-a05e-88b608cc827f\") " pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:02:13.227235 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.227191 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:02:13.230109 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.230087 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kl8n\" (UniqueName: \"kubernetes.io/projected/7863615c-db3b-4db2-976d-22e559411059-kube-api-access-7kl8n\") pod \"network-check-target-8qt8j\" (UID: \"7863615c-db3b-4db2-976d-22e559411059\") " pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:02:13.387182 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.387108 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vm4f" Jul 29 14:02:13.396163 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.396139 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6wqn8\"" Jul 29 14:02:13.401921 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.401900 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7g4zm\"" Jul 29 14:02:13.404648 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.404621 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:02:13.410300 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:13.410281 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfv5m" Jul 29 14:02:14.035616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.035573 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:02:14.038267 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.038244 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"image-registry-75f56884f8-77qwv\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:02:14.136622 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.136579 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:02:14.139336 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.139305 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7010f78e-9440-452f-9616-8abac941737b-metrics-tls\") pod \"dns-default-sbgpj\" (UID: \"7010f78e-9440-452f-9616-8abac941737b\") " pod="openshift-dns/dns-default-sbgpj" Jul 29 14:02:14.253798 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.253756 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fnsd9\"" Jul 29 14:02:14.261775 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.261742 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:02:14.297488 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.297422 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v4vw8\"" Jul 29 14:02:14.305513 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.305494 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sbgpj" Jul 29 14:02:14.819650 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.819614 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:02:14.819821 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.819681 2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:02:14.820013 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:14.819994 2560 scope.go:117] "RemoveContainer" containerID="9f24da733116b8cb8be85b2f4808d743cbcb1c2b1faa1737323c15aa48f4c83f" Jul 29 14:02:14.820207 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:14.820185 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-zn2dp_openshift-console-operator(b63ed94d-a128-4c71-8465-479c86c3cc34)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" podUID="b63ed94d-a128-4c71-8465-479c86c3cc34" Jul 29 14:02:15.398169 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:15.398142 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfv5m"] Jul 29 14:02:15.626770 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:15.626736 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379c825d_878f_4066_841c_084c71477f94.slice/crio-88928ee0d961193e0a9639089f9a54eb61c560f4b3080f13e6ca79c7f406557b WatchSource:0}: Error finding container 88928ee0d961193e0a9639089f9a54eb61c560f4b3080f13e6ca79c7f406557b: Status 404 returned error can't find the container with id 88928ee0d961193e0a9639089f9a54eb61c560f4b3080f13e6ca79c7f406557b Jul 29 14:02:15.709441 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:15.709370 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfv5m" event={"ID":"379c825d-878f-4066-841c-084c71477f94","Type":"ContainerStarted","Data":"88928ee0d961193e0a9639089f9a54eb61c560f4b3080f13e6ca79c7f406557b"} Jul 29 14:02:15.784075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:15.784045 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2vm4f"] Jul 29 14:02:15.787261 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:15.787238 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebe845f_c3d6_4eed_a05e_88b608cc827f.slice/crio-d22aa0ebfe1eee3306e856ff4f02d3bd007f2a489b90dc75d7c2c5ceb7a07571 WatchSource:0}: Error finding container d22aa0ebfe1eee3306e856ff4f02d3bd007f2a489b90dc75d7c2c5ceb7a07571: Status 404 returned error can't find the container with id d22aa0ebfe1eee3306e856ff4f02d3bd007f2a489b90dc75d7c2c5ceb7a07571 Jul 29 14:02:16.021409 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.021384 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8qt8j"] Jul 29 14:02:16.024295 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:16.024267 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7863615c_db3b_4db2_976d_22e559411059.slice/crio-f16f5309f268ba83a665792cb4acd1ac7719efac191dbfd2fb6937a4302ae81e WatchSource:0}: Error finding container f16f5309f268ba83a665792cb4acd1ac7719efac191dbfd2fb6937a4302ae81e: Status 404 returned error can't find the container with id f16f5309f268ba83a665792cb4acd1ac7719efac191dbfd2fb6937a4302ae81e Jul 29 14:02:16.025233 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.025210 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sbgpj"] Jul 29 14:02:16.026033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.026010 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75f56884f8-77qwv"] Jul 29 14:02:16.026961 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:16.026937 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e74d63_8e6d_4ccf_85a9_33c75ba7baa0.slice/crio-48b7fbff78610662019de327c38552709bd81b4e434278ad0978c8e1bb1dfb3e WatchSource:0}: Error finding container 48b7fbff78610662019de327c38552709bd81b4e434278ad0978c8e1bb1dfb3e: Status 404 returned error can't find the container with id 48b7fbff78610662019de327c38552709bd81b4e434278ad0978c8e1bb1dfb3e Jul 29 14:02:16.027432 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:16.027413 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7010f78e_9440_452f_9616_8abac941737b.slice/crio-fb8fcea58ab184e46fbf7037ec238dcb49fe580bdbda313648bb44bcca21fb0d WatchSource:0}: Error finding container fb8fcea58ab184e46fbf7037ec238dcb49fe580bdbda313648bb44bcca21fb0d: Status 404 returned error can't find the container with id fb8fcea58ab184e46fbf7037ec238dcb49fe580bdbda313648bb44bcca21fb0d Jul 29 14:02:16.356861 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.356780 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:02:16.359911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.359878 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a48e2b71-ac38-43d0-8842-9c4e2c9cd833-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xfxm6\" (UID: \"a48e2b71-ac38-43d0-8842-9c4e2c9cd833\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:02:16.458637 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.457883 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:16.458637 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.457942 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:02:16.458637 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.457981 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:16.459715 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.458858 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e04e6603-8385-4ffb-aa85-ccae91e7658f-service-ca-bundle\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:16.463051 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.463002 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7e9787c-6efa-4add-ab5f-81b01326487a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dbx8n\" (UID: \"d7e9787c-6efa-4add-ab5f-81b01326487a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:02:16.464110 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.464084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e04e6603-8385-4ffb-aa85-ccae91e7658f-metrics-certs\") pod \"router-default-78f77ccbf5-cjvmf\" (UID: \"e04e6603-8385-4ffb-aa85-ccae91e7658f\") " pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:16.544576 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.544433 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gkpwm\"" Jul 29 14:02:16.553522 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.553307 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" Jul 29 14:02:16.635987 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.635724 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8plqs\"" Jul 29 14:02:16.644372 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.644331 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" Jul 29 14:02:16.658962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.658479 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n5l9g\"" Jul 29 14:02:16.666563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.666491 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:16.725851 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.725746 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5b4fk" event={"ID":"1d771142-7766-4ff7-8a28-4b3eea8bdfbd","Type":"ContainerStarted","Data":"97a4e4929d5d9c97cc6a5ceb50890ca73a6b160ef62e55c9283cf6e158ef19d4"} Jul 29 14:02:16.729759 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.728844 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" event={"ID":"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0","Type":"ContainerStarted","Data":"bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae"} Jul 29 14:02:16.729759 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.728887 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" event={"ID":"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0","Type":"ContainerStarted","Data":"48b7fbff78610662019de327c38552709bd81b4e434278ad0978c8e1bb1dfb3e"} Jul 29 14:02:16.729759 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.729695 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:02:16.732844 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.732770 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sbgpj" event={"ID":"7010f78e-9440-452f-9616-8abac941737b","Type":"ContainerStarted","Data":"fb8fcea58ab184e46fbf7037ec238dcb49fe580bdbda313648bb44bcca21fb0d"} Jul 29 14:02:16.734477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.734411 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2vm4f" event={"ID":"6ebe845f-c3d6-4eed-a05e-88b608cc827f","Type":"ContainerStarted","Data":"d22aa0ebfe1eee3306e856ff4f02d3bd007f2a489b90dc75d7c2c5ceb7a07571"} Jul 29 14:02:16.738893 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.737988 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6"] Jul 29 14:02:16.740766 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.740620 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8qt8j" event={"ID":"7863615c-db3b-4db2-976d-22e559411059","Type":"ContainerStarted","Data":"b786abca682f09dcbbab589fa3841c73012d1619824659e057858d3d2806f81b"} Jul 29 14:02:16.740766 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.740661 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8qt8j" event={"ID":"7863615c-db3b-4db2-976d-22e559411059","Type":"ContainerStarted","Data":"f16f5309f268ba83a665792cb4acd1ac7719efac191dbfd2fb6937a4302ae81e"} Jul 29 14:02:16.740766 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.740712 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:02:16.746622 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:16.746592 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48e2b71_ac38_43d0_8842_9c4e2c9cd833.slice/crio-6cafae06caa888c62674c03627c826b37d4eefcc92dc6ee984aa0522dbbe3a29 WatchSource:0}: Error finding container 6cafae06caa888c62674c03627c826b37d4eefcc92dc6ee984aa0522dbbe3a29: Status 404 returned error can't find the container with id 6cafae06caa888c62674c03627c826b37d4eefcc92dc6ee984aa0522dbbe3a29 Jul 29 14:02:16.751396 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.751348 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" podStartSLOduration=68.751331213 podStartE2EDuration="1m8.751331213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:16.747193022 +0000 UTC m=+69.948694195" watchObservedRunningTime="2026-07-29 14:02:16.751331213 +0000 UTC m=+69.952832386" Jul 29 14:02:16.763183 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.763128 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8qt8j" podStartSLOduration=69.763108045 podStartE2EDuration="1m9.763108045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:16.762733258 +0000 UTC m=+69.964234431" watchObservedRunningTime="2026-07-29 14:02:16.763108045 +0000 UTC m=+69.964609216" Jul 29 14:02:16.848753 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.848718 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n"] Jul 29 14:02:16.876675 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:16.876597 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78f77ccbf5-cjvmf"] Jul 29 14:02:16.880960 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:16.880899 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e6603_8385_4ffb_aa85_ccae91e7658f.slice/crio-cb88d15c08af8923faf01e060f5326e70b85b702d4acc6fd5919d0f2583bf945 WatchSource:0}: Error finding container cb88d15c08af8923faf01e060f5326e70b85b702d4acc6fd5919d0f2583bf945: Status 404 returned error can't find the container with id cb88d15c08af8923faf01e060f5326e70b85b702d4acc6fd5919d0f2583bf945 Jul 29 14:02:17.748637 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:17.747843 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" event={"ID":"e04e6603-8385-4ffb-aa85-ccae91e7658f","Type":"ContainerStarted","Data":"22ad3d617c13720ef8c2d67c5781e6e66b890534e65390664693dbad51b5437b"} Jul 29 14:02:17.748637 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:17.747901 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" event={"ID":"e04e6603-8385-4ffb-aa85-ccae91e7658f","Type":"ContainerStarted","Data":"cb88d15c08af8923faf01e060f5326e70b85b702d4acc6fd5919d0f2583bf945"} Jul 29 14:02:17.754044 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:17.753968 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" event={"ID":"d7e9787c-6efa-4add-ab5f-81b01326487a","Type":"ContainerStarted","Data":"8f1c8b2712bf6edd2b476031efaf43a67c653176750257305e1f632503d2b8b8"} Jul 29 14:02:17.757447 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:17.757027 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" event={"ID":"a48e2b71-ac38-43d0-8842-9c4e2c9cd833","Type":"ContainerStarted","Data":"6cafae06caa888c62674c03627c826b37d4eefcc92dc6ee984aa0522dbbe3a29"} Jul 29 14:02:18.667192 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:18.667160 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:18.670619 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:18.670594 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:18.694711 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:18.694646 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" podStartSLOduration=34.694624405 podStartE2EDuration="34.694624405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:17.769263451 +0000 UTC m=+70.970764626" watchObservedRunningTime="2026-07-29 14:02:18.694624405 +0000 UTC m=+71.896125580" Jul 29 14:02:18.762966 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:18.762936 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:18.764037 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:18.763965 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-78f77ccbf5-cjvmf" Jul 29 14:02:24.784579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:24.784529 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfv5m" event={"ID":"379c825d-878f-4066-841c-084c71477f94","Type":"ContainerStarted","Data":"ead7f7d380c27bec5dcb5dfa7ffa60124626a0381dc2a535e33e0a39207089a2"} Jul 29 14:02:24.787758 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:24.787730 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" event={"ID":"a48e2b71-ac38-43d0-8842-9c4e2c9cd833","Type":"ContainerStarted","Data":"d3be8ec30c7b67905a5d47d16b01530434cf4189c00ba58c64e0a02b27ef02c1"} Jul 29 14:02:24.806058 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:24.806004 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xfxm6" podStartSLOduration=33.651110949 podStartE2EDuration="40.80599022s" totalImagesPullingTime="7.154879271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:24.804179892 +0000 UTC m=+78.005681065" watchObservedRunningTime="2026-07-29 14:02:24.80599022 +0000 UTC m=+78.007491392" Jul 29 14:02:25.794114 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.794074 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfv5m" event={"ID":"379c825d-878f-4066-841c-084c71477f94","Type":"ContainerStarted","Data":"888e5b9bf5e6e723f62d6d909bf1e671dcd8029fdd86fc3635cdbce025738993"} Jul 29 14:02:25.796176 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.796149 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5b4fk" event={"ID":"1d771142-7766-4ff7-8a28-4b3eea8bdfbd","Type":"ContainerStarted","Data":"c572111667a853c341487d96d8c2e30879948c6fd47a6da086467b2af9429bae"} Jul 29 14:02:25.798174 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.798148 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sbgpj" event={"ID":"7010f78e-9440-452f-9616-8abac941737b","Type":"ContainerStarted","Data":"fb59557578929f2bb788d37253ddf734633584499e2762fb5c2aa93348dcaf18"} Jul 29 14:02:25.798306 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.798195 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sbgpj" event={"ID":"7010f78e-9440-452f-9616-8abac941737b","Type":"ContainerStarted","Data":"eb060a7fb183610e0843b69e369a78ccfd7f32f11617666ed0017a64b89dfa35"} Jul 29 14:02:25.798306 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.798228 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sbgpj" Jul 29 14:02:25.800118 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.800094 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2vm4f" event={"ID":"6ebe845f-c3d6-4eed-a05e-88b608cc827f","Type":"ContainerStarted","Data":"dc7d7fd1d2d59e27203d51ca4757af894f866510bbfb78414988cf731c2f079d"} Jul 29 14:02:25.802106 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.802083 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" event={"ID":"d7e9787c-6efa-4add-ab5f-81b01326487a","Type":"ContainerStarted","Data":"6db650eac26132821080098e2955eb8060879a8bab4dfd90c027a2f962135fda"} Jul 29 14:02:25.802210 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.802128 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" event={"ID":"d7e9787c-6efa-4add-ab5f-81b01326487a","Type":"ContainerStarted","Data":"85e027b6004c5372c2981006684ef88688ba6b84f78bc86ff1c0efaed193d9ff"} Jul 29 14:02:25.814663 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.814626 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vfv5m" podStartSLOduration=69.53626137 podStartE2EDuration="1m17.814614085s" totalImagesPullingTime="8.278352715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:25.814123863 +0000 UTC m=+79.015625036" watchObservedRunningTime="2026-07-29 14:02:25.814614085 +0000 UTC m=+79.016115256" Jul 29 14:02:25.828842 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.828793 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2vm4f" podStartSLOduration=69.698990135 podStartE2EDuration="1m17.828780251s" totalImagesPullingTime="8.129790116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:25.827283224 +0000 UTC m=+79.028784409" watchObservedRunningTime="2026-07-29 14:02:25.828780251 +0000 UTC m=+79.030281423" Jul 29 14:02:25.842430 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.842377 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sbgpj" podStartSLOduration=34.832377656 podStartE2EDuration="43.842146995s" totalImagesPullingTime="9.009769339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:25.841873809 +0000 UTC m=+79.043374983" watchObservedRunningTime="2026-07-29 14:02:25.842146995 +0000 UTC m=+79.043648168" Jul 29 14:02:25.861212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:25.860415 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dbx8n" podStartSLOduration=34.874002917 podStartE2EDuration="41.860390819s" totalImagesPullingTime="6.986387902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:25.856764547 +0000 UTC m=+79.058265720" watchObservedRunningTime="2026-07-29 14:02:25.860390819 +0000 UTC m=+79.061891992" Jul 29 14:02:28.374274 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:28.374246 2560 scope.go:117] "RemoveContainer" containerID="9f24da733116b8cb8be85b2f4808d743cbcb1c2b1faa1737323c15aa48f4c83f" Jul 29 14:02:28.374654 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:28.374392 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-zn2dp_openshift-console-operator(b63ed94d-a128-4c71-8465-479c86c3cc34)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" podUID="b63ed94d-a128-4c71-8465-479c86c3cc34" Jul 29 14:02:28.815601 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:28.815564 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5b4fk" event={"ID":"1d771142-7766-4ff7-8a28-4b3eea8bdfbd","Type":"ContainerStarted","Data":"90da03072f9d26d771517cdc2d9171b3135c9d252726ac84ea5fb0e708f70d18"} Jul 29 14:02:28.832639 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:28.832585 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5b4fk" podStartSLOduration=17.644258408 podStartE2EDuration="34.832569897s" totalImagesPullingTime="17.188311489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:01:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:28.83185718 +0000 UTC m=+82.033358331" watchObservedRunningTime="2026-07-29 14:02:28.832569897 +0000 UTC m=+82.034071098" Jul 29 14:02:34.465676 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.465640 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6xvf2"] Jul 29 14:02:34.468357 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.468341 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.470677 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.470655 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 29 14:02:34.471442 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.471426 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 29 14:02:34.471442 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.471436 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 14:02:34.471600 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.471461 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-g7z99\"" Jul 29 14:02:34.478321 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.478300 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6xvf2"] Jul 29 14:02:34.602811 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.602781 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48b47b5f-c165-4452-b079-5c96b6a7fb3f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.602969 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.602816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpcg5\" (UniqueName: \"kubernetes.io/projected/48b47b5f-c165-4452-b079-5c96b6a7fb3f-kube-api-access-fpcg5\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.602969 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.602864 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48b47b5f-c165-4452-b079-5c96b6a7fb3f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.602969 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.602890 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b47b5f-c165-4452-b079-5c96b6a7fb3f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.703439 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.703409 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48b47b5f-c165-4452-b079-5c96b6a7fb3f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.703439 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.703441 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpcg5\" (UniqueName: \"kubernetes.io/projected/48b47b5f-c165-4452-b079-5c96b6a7fb3f-kube-api-access-fpcg5\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.703715 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.703479 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48b47b5f-c165-4452-b079-5c96b6a7fb3f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.703715 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.703581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b47b5f-c165-4452-b079-5c96b6a7fb3f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.704204 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.704179 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48b47b5f-c165-4452-b079-5c96b6a7fb3f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.705992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.705968 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48b47b5f-c165-4452-b079-5c96b6a7fb3f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.706110 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.706080 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b47b5f-c165-4452-b079-5c96b6a7fb3f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.711394 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.711357 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpcg5\" (UniqueName: \"kubernetes.io/projected/48b47b5f-c165-4452-b079-5c96b6a7fb3f-kube-api-access-fpcg5\") pod \"prometheus-operator-84d69f794d-6xvf2\" (UID: \"48b47b5f-c165-4452-b079-5c96b6a7fb3f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.777075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.777010 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" Jul 29 14:02:34.907910 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:34.907762 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6xvf2"] Jul 29 14:02:34.910528 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:34.910501 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b47b5f_c165_4452_b079_5c96b6a7fb3f.slice/crio-250ea940278b2446c2a7be390940a65244da56c9bb1cf941541d112bcb03eba5 WatchSource:0}: Error finding container 250ea940278b2446c2a7be390940a65244da56c9bb1cf941541d112bcb03eba5: Status 404 returned error can't find the container with id 250ea940278b2446c2a7be390940a65244da56c9bb1cf941541d112bcb03eba5 Jul 29 14:02:35.809777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:35.809748 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sbgpj" Jul 29 14:02:35.845306 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:35.845265 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" event={"ID":"48b47b5f-c165-4452-b079-5c96b6a7fb3f","Type":"ContainerStarted","Data":"250ea940278b2446c2a7be390940a65244da56c9bb1cf941541d112bcb03eba5"} Jul 29 14:02:36.850817 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:36.850780 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" event={"ID":"48b47b5f-c165-4452-b079-5c96b6a7fb3f","Type":"ContainerStarted","Data":"98b140410601496a36c954e7c7238669c68b4e58a564e6e2889eeedf33387ace"} Jul 29 14:02:36.851273 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:36.850870 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" event={"ID":"48b47b5f-c165-4452-b079-5c96b6a7fb3f","Type":"ContainerStarted","Data":"4e9b77fbe922a91f5115e0a9348ef82ce5f89e0ea7573d5a5f9d0b7fbcce78a8"} Jul 29 14:02:36.867888 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:36.867843 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-6xvf2" podStartSLOduration=1.468414903 podStartE2EDuration="2.86781773s" totalImagesPullingTime="1.399402827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:36.866712566 +0000 UTC m=+90.068213738" watchObservedRunningTime="2026-07-29 14:02:36.86781773 +0000 UTC m=+90.069318902" Jul 29 14:02:38.767439 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.767415 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:02:38.838874 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.838840 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lbzn6"] Jul 29 14:02:38.841288 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.841271 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:38.848635 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.848566 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vrdkg\"" Jul 29 14:02:38.848887 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.848874 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 14:02:38.848945 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.848889 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 14:02:38.849167 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.849155 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 14:02:38.936787 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.936752 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26sz\" (UniqueName: \"kubernetes.io/projected/67b41abc-c4f0-4424-8eec-aaebbfec736a-kube-api-access-t26sz\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:38.936985 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.936794 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b41abc-c4f0-4424-8eec-aaebbfec736a-metrics-client-ca\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:38.936985 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.936928 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-accelerators-collector-config\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:38.937143 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.937086 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-tls\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:38.937199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.937157 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-wtmp\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:38.937199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.937186 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67b41abc-c4f0-4424-8eec-aaebbfec736a-root\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:38.937298 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.937208 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-textfile\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:38.937357 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.937332 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67b41abc-c4f0-4424-8eec-aaebbfec736a-sys\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:38.937411 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:38.937360 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038314 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038225 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67b41abc-c4f0-4424-8eec-aaebbfec736a-sys\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038314 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038266 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038345 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67b41abc-c4f0-4424-8eec-aaebbfec736a-sys\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038400 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26sz\" (UniqueName: \"kubernetes.io/projected/67b41abc-c4f0-4424-8eec-aaebbfec736a-kube-api-access-t26sz\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038428 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b41abc-c4f0-4424-8eec-aaebbfec736a-metrics-client-ca\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038457 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-accelerators-collector-config\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038495 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-tls\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038532 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-wtmp\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67b41abc-c4f0-4424-8eec-aaebbfec736a-root\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038951 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-textfile\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038951 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038712 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-wtmp\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038951 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038769 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67b41abc-c4f0-4424-8eec-aaebbfec736a-root\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.038951 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.038940 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-textfile\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.039321 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.039103 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b41abc-c4f0-4424-8eec-aaebbfec736a-metrics-client-ca\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.039321 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.039111 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-accelerators-collector-config\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.040836 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.040816 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.040950 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.040935 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67b41abc-c4f0-4424-8eec-aaebbfec736a-node-exporter-tls\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.046750 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.046727 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26sz\" (UniqueName: \"kubernetes.io/projected/67b41abc-c4f0-4424-8eec-aaebbfec736a-kube-api-access-t26sz\") pod \"node-exporter-lbzn6\" (UID: \"67b41abc-c4f0-4424-8eec-aaebbfec736a\") " pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.149839 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.149798 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lbzn6" Jul 29 14:02:39.160362 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:39.160337 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b41abc_c4f0_4424_8eec_aaebbfec736a.slice/crio-8b9b974ab84ae93203cd2658fa93962a31261a7bd03ef681cc6d23aa035606b9 WatchSource:0}: Error finding container 8b9b974ab84ae93203cd2658fa93962a31261a7bd03ef681cc6d23aa035606b9: Status 404 returned error can't find the container with id 8b9b974ab84ae93203cd2658fa93962a31261a7bd03ef681cc6d23aa035606b9 Jul 29 14:02:39.867104 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.867066 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lbzn6" event={"ID":"67b41abc-c4f0-4424-8eec-aaebbfec736a","Type":"ContainerStarted","Data":"8b9b974ab84ae93203cd2658fa93962a31261a7bd03ef681cc6d23aa035606b9"} Jul 29 14:02:39.918373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.918332 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 14:02:39.921612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.921589 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:39.924137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.923882 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 14:02:39.924137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.923964 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 14:02:39.924137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.924013 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 14:02:39.924137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.924030 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 14:02:39.924137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.923895 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 14:02:39.924468 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.924227 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 14:02:39.924877 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.924725 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 14:02:39.924877 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.924785 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 14:02:39.925049 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.924999 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 14:02:39.925269 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.925248 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wtqrm\"" Jul 29 14:02:39.935829 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:39.935808 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 14:02:40.047773 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.047744 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.047962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.047783 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-config-out\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.047962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.047807 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75jb6\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-kube-api-access-75jb6\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.047962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.047849 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.047962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.047879 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.047962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.047900 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.047962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.047915 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.047962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.047936 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.048270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.048024 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.048270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.048065 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-web-config\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.048270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.048091 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.048270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.048115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.048270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.048197 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.148909 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.148809 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75jb6\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-kube-api-access-75jb6\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.148909 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.148868 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.148909 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.148903 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.148937 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.148961 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.148993 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.149026 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.149063 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-web-config\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.149089 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.149111 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.149144 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.149169 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.149633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.149203 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-config-out\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.151254 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:40.150784 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-trusted-ca-bundle podName:1b80753c-28d4-4885-bc7c-604b946d8b78 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:40.650760894 +0000 UTC m=+93.852262057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78") : configmap references non-existent config key: ca-bundle.crt Jul 29 14:02:40.152023 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.151558 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.152023 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.151675 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.153440 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.153416 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.154946 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.154679 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-web-config\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.154946 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.154706 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.154946 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.154754 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-config-volume\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.154946 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.154906 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-config-out\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.155289 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.155257 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.155401 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.155378 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.156006 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.155978 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.156193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.156175 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.164848 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.164822 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75jb6\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-kube-api-access-75jb6\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.654391 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.654138 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.655132 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.655104 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.833985 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.833951 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:02:40.987064 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:40.987039 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b80753c_28d4_4885_bc7c_604b946d8b78.slice/crio-d2bbe16fe95ce4402cc773bfa607845ad91a5928bc5587215c42742c6d8da648 WatchSource:0}: Error finding container d2bbe16fe95ce4402cc773bfa607845ad91a5928bc5587215c42742c6d8da648: Status 404 returned error can't find the container with id d2bbe16fe95ce4402cc773bfa607845ad91a5928bc5587215c42742c6d8da648 Jul 29 14:02:40.987921 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:40.987896 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 14:02:41.379141 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.379113 2560 scope.go:117] "RemoveContainer" containerID="9f24da733116b8cb8be85b2f4808d743cbcb1c2b1faa1737323c15aa48f4c83f" Jul 29 14:02:41.837329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.837292 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-558b99f77-772nh"] Jul 29 14:02:41.842300 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.842273 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:41.846585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.845935 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cec6n28dcfpae\"" Jul 29 14:02:41.846585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.846062 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bvqv4\"" Jul 29 14:02:41.846585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.846162 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 29 14:02:41.846585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.846272 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 29 14:02:41.846585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.845938 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 29 14:02:41.846585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.846436 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 29 14:02:41.846585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.846469 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 29 14:02:41.852009 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.851985 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-558b99f77-772nh"] Jul 29 14:02:41.877029 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.876988 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerStarted","Data":"d2bbe16fe95ce4402cc773bfa607845ad91a5928bc5587215c42742c6d8da648"} Jul 29 14:02:41.879590 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.879565 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:02:41.879729 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.879615 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" event={"ID":"b63ed94d-a128-4c71-8465-479c86c3cc34","Type":"ContainerStarted","Data":"3e5b1ea5bf5dc15ff191a312ee721da730ba4292d6a29d5aa6f4b0947878d150"} Jul 29 14:02:41.880063 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.880046 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:02:41.901867 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.901815 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" podStartSLOduration=51.14783914 podStartE2EDuration="57.901796054s" totalImagesPullingTime="6.753956914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:41.900984194 +0000 UTC m=+95.102485367" watchObservedRunningTime="2026-07-29 14:02:41.901796054 +0000 UTC m=+95.103297228" Jul 29 14:02:41.967528 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.967487 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:41.967716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.967566 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:41.967716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.967596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-grpc-tls\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:41.967716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.967616 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:41.967716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.967694 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:41.967901 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.967736 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzbv\" (UniqueName: \"kubernetes.io/projected/bef742fe-a63b-4f80-bfd5-0435ce451118-kube-api-access-jjzbv\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:41.967901 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.967769 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef742fe-a63b-4f80-bfd5-0435ce451118-metrics-client-ca\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:41.967901 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:41.967807 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-tls\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.068586 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.068549 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.069031 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.068602 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.069031 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.068621 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-grpc-tls\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.069031 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.068648 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.069031 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.068680 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.069031 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.068707 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzbv\" (UniqueName: \"kubernetes.io/projected/bef742fe-a63b-4f80-bfd5-0435ce451118-kube-api-access-jjzbv\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.069031 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.068726 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef742fe-a63b-4f80-bfd5-0435ce451118-metrics-client-ca\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.069031 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.068763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-tls\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.071690 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.070381 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef742fe-a63b-4f80-bfd5-0435ce451118-metrics-client-ca\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.071988 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.071959 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.072400 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.072379 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-tls\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.072988 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.072969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-grpc-tls\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.073059 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.072995 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.073059 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.072976 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.074204 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.074183 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef742fe-a63b-4f80-bfd5-0435ce451118-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.079125 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.079100 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzbv\" (UniqueName: \"kubernetes.io/projected/bef742fe-a63b-4f80-bfd5-0435ce451118-kube-api-access-jjzbv\") pod \"thanos-querier-558b99f77-772nh\" (UID: \"bef742fe-a63b-4f80-bfd5-0435ce451118\") " pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.118490 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.118406 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-zn2dp" Jul 29 14:02:42.155605 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.155532 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:42.351996 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.351967 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75f56884f8-77qwv"] Jul 29 14:02:42.496150 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.496092 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-558b99f77-772nh"] Jul 29 14:02:42.517472 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:42.517374 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef742fe_a63b_4f80_bfd5_0435ce451118.slice/crio-7354db3f7eb9d58a07b742b4a069c609c05868d1e408c75db6a79bfe68e51b39 WatchSource:0}: Error finding container 7354db3f7eb9d58a07b742b4a069c609c05868d1e408c75db6a79bfe68e51b39: Status 404 returned error can't find the container with id 7354db3f7eb9d58a07b742b4a069c609c05868d1e408c75db6a79bfe68e51b39 Jul 29 14:02:42.885012 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.884970 2560 generic.go:356] "Generic (PLEG): container finished" podID="67b41abc-c4f0-4424-8eec-aaebbfec736a" containerID="9c76b76a1aea64473c72d80caa87123434a12db7c90b4ce10a8d4b53cae3b5a5" exitCode=0 Jul 29 14:02:42.885191 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.885057 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lbzn6" event={"ID":"67b41abc-c4f0-4424-8eec-aaebbfec736a","Type":"ContainerDied","Data":"9c76b76a1aea64473c72d80caa87123434a12db7c90b4ce10a8d4b53cae3b5a5"} Jul 29 14:02:42.886348 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.886321 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" event={"ID":"bef742fe-a63b-4f80-bfd5-0435ce451118","Type":"ContainerStarted","Data":"7354db3f7eb9d58a07b742b4a069c609c05868d1e408c75db6a79bfe68e51b39"} Jul 29 14:02:42.887882 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.887860 2560 generic.go:356] "Generic (PLEG): container finished" podID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerID="20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9" exitCode=0 Jul 29 14:02:42.887971 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:42.887935 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerDied","Data":"20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9"} Jul 29 14:02:43.587719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.587683 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb"] Jul 29 14:02:43.591625 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.591602 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:43.593826 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.593803 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 29 14:02:43.593953 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.593807 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lwjj7\"" Jul 29 14:02:43.600667 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.600646 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb"] Jul 29 14:02:43.784760 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.784721 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmqp\" (UniqueName: \"kubernetes.io/projected/9e896d9b-f6a9-4f21-b9f0-e05deb1315c0-kube-api-access-tbmqp\") pod \"monitoring-plugin-68b7fc9574-2ghhb\" (UID: \"9e896d9b-f6a9-4f21-b9f0-e05deb1315c0\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:43.784929 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.784790 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e896d9b-f6a9-4f21-b9f0-e05deb1315c0-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2ghhb\" (UID: \"9e896d9b-f6a9-4f21-b9f0-e05deb1315c0\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:43.885419 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.885380 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e896d9b-f6a9-4f21-b9f0-e05deb1315c0-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2ghhb\" (UID: \"9e896d9b-f6a9-4f21-b9f0-e05deb1315c0\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:43.885645 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.885490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmqp\" (UniqueName: \"kubernetes.io/projected/9e896d9b-f6a9-4f21-b9f0-e05deb1315c0-kube-api-access-tbmqp\") pod \"monitoring-plugin-68b7fc9574-2ghhb\" (UID: \"9e896d9b-f6a9-4f21-b9f0-e05deb1315c0\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:43.885984 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:43.885962 2560 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 29 14:02:43.886054 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:02:43.886032 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e896d9b-f6a9-4f21-b9f0-e05deb1315c0-monitoring-plugin-cert podName:9e896d9b-f6a9-4f21-b9f0-e05deb1315c0 nodeName:}" failed. No retries permitted until 2026-07-29 14:02:44.386011976 +0000 UTC m=+97.587513133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9e896d9b-f6a9-4f21-b9f0-e05deb1315c0-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-2ghhb" (UID: "9e896d9b-f6a9-4f21-b9f0-e05deb1315c0") : secret "monitoring-plugin-cert" not found Jul 29 14:02:43.895073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.895047 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmqp\" (UniqueName: \"kubernetes.io/projected/9e896d9b-f6a9-4f21-b9f0-e05deb1315c0-kube-api-access-tbmqp\") pod \"monitoring-plugin-68b7fc9574-2ghhb\" (UID: \"9e896d9b-f6a9-4f21-b9f0-e05deb1315c0\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:43.899251 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.899224 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lbzn6" event={"ID":"67b41abc-c4f0-4424-8eec-aaebbfec736a","Type":"ContainerStarted","Data":"37d8d7196aba78604c152943112e5f7ca3e84597680384d16a082ae98afc323d"} Jul 29 14:02:43.899384 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:43.899267 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lbzn6" event={"ID":"67b41abc-c4f0-4424-8eec-aaebbfec736a","Type":"ContainerStarted","Data":"ef27ed4dfbf3880476668b19f887f3ebd27b570c24c065bf0a970f8ffad7b487"} Jul 29 14:02:44.391685 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:44.391648 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e896d9b-f6a9-4f21-b9f0-e05deb1315c0-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2ghhb\" (UID: \"9e896d9b-f6a9-4f21-b9f0-e05deb1315c0\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:44.398818 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:44.398788 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e896d9b-f6a9-4f21-b9f0-e05deb1315c0-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2ghhb\" (UID: \"9e896d9b-f6a9-4f21-b9f0-e05deb1315c0\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:44.503014 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:44.502976 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:45.489241 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.489124 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lbzn6" podStartSLOduration=4.219624004 podStartE2EDuration="7.489102317s" totalImagesPullingTime="3.194921631s" totalInitContainerRuntime="74.556682ms" isStatefulPod=true podCreationTimestamp="2026-07-29 14:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:43.918970185 +0000 UTC m=+97.120471359" watchObservedRunningTime="2026-07-29 14:02:45.489102317 +0000 UTC m=+98.690603490" Jul 29 14:02:45.490139 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.490110 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb"] Jul 29 14:02:45.492988 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:02:45.492958 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e896d9b_f6a9_4f21_b9f0_e05deb1315c0.slice/crio-07401aa1bed5a110db3b057e464daa17e0c58fc0428e14c78236fc5dd5276c11 WatchSource:0}: Error finding container 07401aa1bed5a110db3b057e464daa17e0c58fc0428e14c78236fc5dd5276c11: Status 404 returned error can't find the container with id 07401aa1bed5a110db3b057e464daa17e0c58fc0428e14c78236fc5dd5276c11 Jul 29 14:02:45.910633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.910510 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" event={"ID":"bef742fe-a63b-4f80-bfd5-0435ce451118","Type":"ContainerStarted","Data":"c7f8c1073982859286dce9d8f4de8364dc3bd1f338cebc1d1b4a4e4226caa3c5"} Jul 29 14:02:45.910633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.910615 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" event={"ID":"bef742fe-a63b-4f80-bfd5-0435ce451118","Type":"ContainerStarted","Data":"0ca9a1dc32a97932041ede5e6b0464ba55cc07c1bfb508bf12f93168e9b17897"} Jul 29 14:02:45.910838 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.910658 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" event={"ID":"bef742fe-a63b-4f80-bfd5-0435ce451118","Type":"ContainerStarted","Data":"0c9d9fbee3cc1b976827bf39699bdace4b1676200ecf1cf5991ba65feaddaa72"} Jul 29 14:02:45.911747 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.911723 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" event={"ID":"9e896d9b-f6a9-4f21-b9f0-e05deb1315c0","Type":"ContainerStarted","Data":"07401aa1bed5a110db3b057e464daa17e0c58fc0428e14c78236fc5dd5276c11"} Jul 29 14:02:45.914637 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.914616 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerStarted","Data":"b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b"} Jul 29 14:02:45.914722 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.914666 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerStarted","Data":"78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3"} Jul 29 14:02:45.914722 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.914689 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerStarted","Data":"8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea"} Jul 29 14:02:45.914722 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.914715 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerStarted","Data":"d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423"} Jul 29 14:02:45.914816 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:45.914737 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerStarted","Data":"93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7"} Jul 29 14:02:47.760795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:47.760768 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8qt8j" Jul 29 14:02:47.928272 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:47.928241 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" event={"ID":"bef742fe-a63b-4f80-bfd5-0435ce451118","Type":"ContainerStarted","Data":"0889350df2efd64a8107bc0d33ec1dc665ddd32cecddb562727f7359cbc1cfd9"} Jul 29 14:02:47.929818 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:47.929779 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" event={"ID":"9e896d9b-f6a9-4f21-b9f0-e05deb1315c0","Type":"ContainerStarted","Data":"34507ddaac6e8c8c31eef8055a57ed8dbc8738e123f639a12156f7ef432f273e"} Jul 29 14:02:47.932791 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:47.932770 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerStarted","Data":"723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0"} Jul 29 14:02:47.953919 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:47.953789 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:47.958642 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:47.958615 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" Jul 29 14:02:47.970146 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:47.970107 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2ghhb" podStartSLOduration=2.7061123990000002 podStartE2EDuration="4.970092762s" totalImagesPullingTime="2.263980363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:47.96969017 +0000 UTC m=+101.171191343" watchObservedRunningTime="2026-07-29 14:02:47.970092762 +0000 UTC m=+101.171593911" Jul 29 14:02:48.019351 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:48.019305 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.088488347 podStartE2EDuration="9.019287745s" totalImagesPullingTime="5.914824433s" totalInitContainerRuntime="15.974965ms" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:48.017490255 +0000 UTC m=+101.218991427" watchObservedRunningTime="2026-07-29 14:02:48.019287745 +0000 UTC m=+101.220788917" Jul 29 14:02:48.940080 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:48.940047 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" event={"ID":"bef742fe-a63b-4f80-bfd5-0435ce451118","Type":"ContainerStarted","Data":"e3ef5836f67b90c08239077c856f4c56d0be83eb184be45889ccd1b2d51ff40e"} Jul 29 14:02:48.940489 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:48.940111 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" event={"ID":"bef742fe-a63b-4f80-bfd5-0435ce451118","Type":"ContainerStarted","Data":"0d22b407248b8f0bdf62fc5c35048be89e6110f41b16b2b6de66cd0e664f22b3"} Jul 29 14:02:48.940575 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:48.940557 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:02:48.963377 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:48.963319 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" podStartSLOduration=2.9843371899999998 podStartE2EDuration="7.963304127s" totalImagesPullingTime="4.978966937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:02:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:02:48.960894027 +0000 UTC m=+102.162395202" watchObservedRunningTime="2026-07-29 14:02:48.963304127 +0000 UTC m=+102.164805298" Jul 29 14:02:54.958640 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:02:54.958610 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-558b99f77-772nh" Jul 29 14:03:07.374726 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.374656 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" podUID="07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" containerName="registry" containerID="cri-o://bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae" gracePeriod=30 Jul 29 14:03:07.620120 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.620097 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:03:07.693446 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.693413 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-ca-trust-extracted\") pod \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " Jul 29 14:03:07.693446 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.693449 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktjzz\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-kube-api-access-ktjzz\") pod \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " Jul 29 14:03:07.693727 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.693472 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-bound-sa-token\") pod \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " Jul 29 14:03:07.693727 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.693501 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-certificates\") pod \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " Jul 29 14:03:07.693727 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.693659 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-trusted-ca\") pod \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " Jul 29 14:03:07.693887 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.693729 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-installation-pull-secrets\") pod \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " Jul 29 14:03:07.693887 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.693774 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-image-registry-private-configuration\") pod \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " Jul 29 14:03:07.693887 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.693800 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") pod \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\" (UID: \"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0\") " Jul 29 14:03:07.694086 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.694055 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:03:07.694344 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.694300 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:03:07.696533 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.696499 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:03:07.696681 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.696664 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-kube-api-access-ktjzz" (OuterVolumeSpecName: "kube-api-access-ktjzz") pod "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0"). InnerVolumeSpecName "kube-api-access-ktjzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:03:07.696775 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.696748 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:03:07.696827 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.696783 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:03:07.696866 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.696822 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:03:07.702424 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.702402 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" (UID: "07e74d63-8e6d-4ccf-85a9-33c75ba7baa0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:03:07.794670 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.794636 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-image-registry-private-configuration\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:03:07.794670 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.794663 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-tls\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:03:07.794670 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.794674 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-ca-trust-extracted\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:03:07.794879 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.794682 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktjzz\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-kube-api-access-ktjzz\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:03:07.794879 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.794692 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-bound-sa-token\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:03:07.794879 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.794700 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-registry-certificates\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:03:07.794879 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.794708 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-trusted-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:03:07.794879 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:07.794715 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0-installation-pull-secrets\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:03:08.026823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:08.026730 2560 generic.go:356] "Generic (PLEG): container finished" podID="07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" containerID="bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae" exitCode=0 Jul 29 14:03:08.026823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:08.026779 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" Jul 29 14:03:08.026823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:08.026813 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" event={"ID":"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0","Type":"ContainerDied","Data":"bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae"} Jul 29 14:03:08.027037 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:08.026861 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f56884f8-77qwv" event={"ID":"07e74d63-8e6d-4ccf-85a9-33c75ba7baa0","Type":"ContainerDied","Data":"48b7fbff78610662019de327c38552709bd81b4e434278ad0978c8e1bb1dfb3e"} Jul 29 14:03:08.027037 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:08.026881 2560 scope.go:117] "RemoveContainer" containerID="bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae" Jul 29 14:03:08.035678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:08.035618 2560 scope.go:117] "RemoveContainer" containerID="bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae" Jul 29 14:03:08.035912 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:03:08.035888 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae\": container with ID starting with bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae not found: ID does not exist" containerID="bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae" Jul 29 14:03:08.035995 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:08.035919 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae"} err="failed to get container status \"bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae\": rpc error: code = NotFound desc = could not find container \"bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae\": container with ID starting with bbb331904887632c9c983622cfd09024a31fdb58d40c69986b5eb37d492780ae not found: ID does not exist" Jul 29 14:03:08.047323 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:08.047302 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75f56884f8-77qwv"] Jul 29 14:03:08.050932 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:08.050901 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75f56884f8-77qwv"] Jul 29 14:03:09.032354 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:09.032321 2560 generic.go:356] "Generic (PLEG): container finished" podID="58f1ae35-67f6-46a3-9a41-7cd808fe739c" containerID="5a5b0aa6f0ee512c14e37fb07746c2b37162119a0b586decb01f40f619bb315d" exitCode=0 Jul 29 14:03:09.032830 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:09.032396 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-pt4nq" event={"ID":"58f1ae35-67f6-46a3-9a41-7cd808fe739c","Type":"ContainerDied","Data":"5a5b0aa6f0ee512c14e37fb07746c2b37162119a0b586decb01f40f619bb315d"} Jul 29 14:03:09.032830 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:09.032797 2560 scope.go:117] "RemoveContainer" containerID="5a5b0aa6f0ee512c14e37fb07746c2b37162119a0b586decb01f40f619bb315d" Jul 29 14:03:09.379461 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:09.379380 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" path="/var/lib/kubelet/pods/07e74d63-8e6d-4ccf-85a9-33c75ba7baa0/volumes" Jul 29 14:03:10.044091 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:10.044022 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-pt4nq" event={"ID":"58f1ae35-67f6-46a3-9a41-7cd808fe739c","Type":"ContainerStarted","Data":"220fbbbcc887d08f59d472c6557f281abc3a16f4305138eb9de8d05ad9811911"} Jul 29 14:03:10.529765 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:10.529740 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b80753c-28d4-4885-bc7c-604b946d8b78/init-config-reloader/0.log" Jul 29 14:03:10.730989 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:10.730952 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b80753c-28d4-4885-bc7c-604b946d8b78/alertmanager/0.log" Jul 29 14:03:10.929830 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:10.929806 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b80753c-28d4-4885-bc7c-604b946d8b78/config-reloader/0.log" Jul 29 14:03:11.130402 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:11.130376 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b80753c-28d4-4885-bc7c-604b946d8b78/kube-rbac-proxy-web/0.log" Jul 29 14:03:11.339979 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:11.339901 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b80753c-28d4-4885-bc7c-604b946d8b78/kube-rbac-proxy/0.log" Jul 29 14:03:11.530980 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:11.530950 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b80753c-28d4-4885-bc7c-604b946d8b78/kube-rbac-proxy-metric/0.log" Jul 29 14:03:11.730180 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:11.730153 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1b80753c-28d4-4885-bc7c-604b946d8b78/prom-label-proxy/0.log" Jul 29 14:03:11.931910 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:11.931878 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-xfxm6_a48e2b71-ac38-43d0-8842-9c4e2c9cd833/cluster-monitoring-operator/0.log" Jul 29 14:03:12.929867 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:12.929840 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-2ghhb_9e896d9b-f6a9-4f21-b9f0-e05deb1315c0/monitoring-plugin/0.log" Jul 29 14:03:13.729203 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:13.729177 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lbzn6_67b41abc-c4f0-4424-8eec-aaebbfec736a/init-textfile/0.log" Jul 29 14:03:13.931341 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:13.931314 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lbzn6_67b41abc-c4f0-4424-8eec-aaebbfec736a/node-exporter/0.log" Jul 29 14:03:14.129947 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:14.129869 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lbzn6_67b41abc-c4f0-4424-8eec-aaebbfec736a/kube-rbac-proxy/0.log" Jul 29 14:03:16.932917 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:16.932882 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6xvf2_48b47b5f-c165-4452-b079-5c96b6a7fb3f/prometheus-operator/0.log" Jul 29 14:03:17.130815 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:17.130780 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6xvf2_48b47b5f-c165-4452-b079-5c96b6a7fb3f/kube-rbac-proxy/0.log" Jul 29 14:03:18.078089 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:18.078052 2560 generic.go:356] "Generic (PLEG): container finished" podID="7d5798e2-d0d9-4635-9dfa-f892e92bfc9f" containerID="7a0e210d3121454188e5af7edd5bcca59cc0f9ad272793bef3e72240fdb7b34b" exitCode=0 Jul 29 14:03:18.078089 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:18.078090 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" event={"ID":"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f","Type":"ContainerDied","Data":"7a0e210d3121454188e5af7edd5bcca59cc0f9ad272793bef3e72240fdb7b34b"} Jul 29 14:03:18.078484 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:18.078393 2560 scope.go:117] "RemoveContainer" containerID="7a0e210d3121454188e5af7edd5bcca59cc0f9ad272793bef3e72240fdb7b34b" Jul 29 14:03:18.129809 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:18.129784 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/thanos-query/0.log" Jul 29 14:03:18.329768 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:18.329694 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/kube-rbac-proxy-web/0.log" Jul 29 14:03:18.530467 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:18.530437 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/kube-rbac-proxy/0.log" Jul 29 14:03:18.729659 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:18.729632 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/prom-label-proxy/0.log" Jul 29 14:03:18.932859 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:18.932830 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/kube-rbac-proxy-rules/0.log" Jul 29 14:03:19.089993 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:19.089915 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q9gzv" event={"ID":"7d5798e2-d0d9-4635-9dfa-f892e92bfc9f","Type":"ContainerStarted","Data":"35db2e5004f2fad0b9d3465e3fe244e7b4aeec465572f1f4323630edcf07d10e"} Jul 29 14:03:19.130727 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:19.130698 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/kube-rbac-proxy-metrics/0.log" Jul 29 14:03:19.530110 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:19.530084 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:03:19.732983 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:19.732947 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/3.log" Jul 29 14:03:20.530018 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:20.529966 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78f77ccbf5-cjvmf_e04e6603-8385-4ffb-aa85-ccae91e7658f/router/0.log" Jul 29 14:03:21.130039 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:21.130007 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nm2x6_0c6e3418-a206-431f-aad9-c917dd9df3f4/serve-healthcheck-canary/0.log" Jul 29 14:03:59.178252 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:59.178160 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 14:03:59.178777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:59.178469 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="alertmanager" containerID="cri-o://93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7" gracePeriod=120 Jul 29 14:03:59.178777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:59.178559 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy-metric" containerID="cri-o://b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b" gracePeriod=120 Jul 29 14:03:59.178777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:59.178599 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="config-reloader" containerID="cri-o://d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423" gracePeriod=120 Jul 29 14:03:59.178777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:59.178623 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="prom-label-proxy" containerID="cri-o://723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0" gracePeriod=120 Jul 29 14:03:59.178777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:59.178644 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy" containerID="cri-o://78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3" gracePeriod=120 Jul 29 14:03:59.178777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:03:59.178577 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy-web" containerID="cri-o://8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea" gracePeriod=120 Jul 29 14:04:00.258107 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.258081 2560 generic.go:356] "Generic (PLEG): container finished" podID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerID="723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0" exitCode=0 Jul 29 14:04:00.258107 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.258101 2560 generic.go:356] "Generic (PLEG): container finished" podID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerID="78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3" exitCode=0 Jul 29 14:04:00.258107 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.258107 2560 generic.go:356] "Generic (PLEG): container finished" podID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerID="d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423" exitCode=0 Jul 29 14:04:00.258107 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.258113 2560 generic.go:356] "Generic (PLEG): container finished" podID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerID="93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7" exitCode=0 Jul 29 14:04:00.258472 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.258153 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerDied","Data":"723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0"} Jul 29 14:04:00.258472 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.258214 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerDied","Data":"78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3"} Jul 29 14:04:00.258472 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.258243 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerDied","Data":"d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423"} Jul 29 14:04:00.258472 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.258267 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerDied","Data":"93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7"} Jul 29 14:04:00.414489 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.414468 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:00.532079 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.531994 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75jb6\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-kube-api-access-75jb6\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532079 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532030 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532079 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532052 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-metrics-client-ca\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532079 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532077 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532219 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-main-db\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532254 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-cluster-tls-config\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532282 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-web-config\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532315 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-web\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532355 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-config-out\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532384 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-main-tls\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532421 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-trusted-ca-bundle\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532446 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-config-volume\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532488 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-tls-assets\") pod \"1b80753c-28d4-4885-bc7c-604b946d8b78\" (UID: \"1b80753c-28d4-4885-bc7c-604b946d8b78\") " Jul 29 14:04:00.532716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532571 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:04:00.532716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532530 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:04:00.533133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532800 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-metrics-client-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.533133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.532824 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-main-db\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.533229 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.533198 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:04:00.534725 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.534701 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-kube-api-access-75jb6" (OuterVolumeSpecName: "kube-api-access-75jb6") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "kube-api-access-75jb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:04:00.535679 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.535114 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:00.535679 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.535232 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:00.535679 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.535350 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:04:00.536048 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.536008 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:00.536156 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.536107 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-config-out" (OuterVolumeSpecName: "config-out") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:04:00.536911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.536891 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-config-volume" (OuterVolumeSpecName: "config-volume") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:00.537616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.537593 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:00.540026 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.539959 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:00.546305 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.546284 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-web-config" (OuterVolumeSpecName: "web-config") pod "1b80753c-28d4-4885-bc7c-604b946d8b78" (UID: "1b80753c-28d4-4885-bc7c-604b946d8b78"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:04:00.633837 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633805 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75jb6\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-kube-api-access-75jb6\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.633837 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633835 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.634003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633845 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.634003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633855 2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-cluster-tls-config\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.634003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633864 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-web-config\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.634003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633874 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.634003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633882 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b80753c-28d4-4885-bc7c-604b946d8b78-config-out\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.634003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633891 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-secret-alertmanager-main-tls\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.634003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633900 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b80753c-28d4-4885-bc7c-604b946d8b78-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.634003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633908 2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1b80753c-28d4-4885-bc7c-604b946d8b78-config-volume\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:00.634003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:00.633916 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b80753c-28d4-4885-bc7c-604b946d8b78-tls-assets\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:04:01.265370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.265341 2560 generic.go:356] "Generic (PLEG): container finished" podID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerID="b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b" exitCode=0 Jul 29 14:04:01.265370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.265363 2560 generic.go:356] "Generic (PLEG): container finished" podID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerID="8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea" exitCode=0 Jul 29 14:04:01.265763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.265391 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerDied","Data":"b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b"} Jul 29 14:04:01.265763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.265413 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.265763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.265448 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerDied","Data":"8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea"} Jul 29 14:04:01.265763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.265476 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1b80753c-28d4-4885-bc7c-604b946d8b78","Type":"ContainerDied","Data":"d2bbe16fe95ce4402cc773bfa607845ad91a5928bc5587215c42742c6d8da648"} Jul 29 14:04:01.265763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.265505 2560 scope.go:117] "RemoveContainer" containerID="723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0" Jul 29 14:04:01.279225 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.279202 2560 scope.go:117] "RemoveContainer" containerID="b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b" Jul 29 14:04:01.286516 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.286499 2560 scope.go:117] "RemoveContainer" containerID="78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3" Jul 29 14:04:01.290847 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.290824 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 14:04:01.294355 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.294191 2560 scope.go:117] "RemoveContainer" containerID="8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea" Jul 29 14:04:01.295675 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.295614 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 14:04:01.301032 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.301017 2560 scope.go:117] "RemoveContainer" containerID="d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423" Jul 29 14:04:01.307473 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.307451 2560 scope.go:117] "RemoveContainer" containerID="93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7" Jul 29 14:04:01.314369 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.314352 2560 scope.go:117] "RemoveContainer" containerID="20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9" Jul 29 14:04:01.321242 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.321228 2560 scope.go:117] "RemoveContainer" containerID="723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0" Jul 29 14:04:01.321468 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:04:01.321451 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0\": container with ID starting with 723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0 not found: ID does not exist" containerID="723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0" Jul 29 14:04:01.321513 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.321477 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0"} err="failed to get container status \"723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0\": rpc error: code = NotFound desc = could not find container \"723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0\": container with ID starting with 723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0 not found: ID does not exist" Jul 29 14:04:01.321513 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.321498 2560 scope.go:117] "RemoveContainer" containerID="b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b" Jul 29 14:04:01.321740 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:04:01.321721 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b\": container with ID starting with b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b not found: ID does not exist" containerID="b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b" Jul 29 14:04:01.321804 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.321749 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b"} err="failed to get container status \"b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b\": rpc error: code = NotFound desc = could not find container \"b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b\": container with ID starting with b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b not found: ID does not exist" Jul 29 14:04:01.321804 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.321772 2560 scope.go:117] "RemoveContainer" containerID="78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3" Jul 29 14:04:01.322007 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:04:01.321993 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3\": container with ID starting with 78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3 not found: ID does not exist" containerID="78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3" Jul 29 14:04:01.322043 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.322010 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3"} err="failed to get container status \"78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3\": rpc error: code = NotFound desc = could not find container \"78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3\": container with ID starting with 78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3 not found: ID does not exist" Jul 29 14:04:01.322043 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.322026 2560 scope.go:117] "RemoveContainer" containerID="8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea" Jul 29 14:04:01.322458 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:04:01.322424 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea\": container with ID starting with 8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea not found: ID does not exist" containerID="8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea" Jul 29 14:04:01.322617 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.322473 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea"} err="failed to get container status \"8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea\": rpc error: code = NotFound desc = could not find container \"8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea\": container with ID starting with 8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea not found: ID does not exist" Jul 29 14:04:01.322617 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.322492 2560 scope.go:117] "RemoveContainer" containerID="d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423" Jul 29 14:04:01.322906 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:04:01.322884 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423\": container with ID starting with d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423 not found: ID does not exist" containerID="d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423" Jul 29 14:04:01.322991 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.322912 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423"} err="failed to get container status \"d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423\": rpc error: code = NotFound desc = could not find container \"d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423\": container with ID starting with d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423 not found: ID does not exist" Jul 29 14:04:01.322991 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.322931 2560 scope.go:117] "RemoveContainer" containerID="93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7" Jul 29 14:04:01.323227 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:04:01.323181 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7\": container with ID starting with 93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7 not found: ID does not exist" containerID="93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7" Jul 29 14:04:01.323227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.323205 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7"} err="failed to get container status \"93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7\": rpc error: code = NotFound desc = could not find container \"93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7\": container with ID starting with 93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7 not found: ID does not exist" Jul 29 14:04:01.323227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.323225 2560 scope.go:117] "RemoveContainer" containerID="20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9" Jul 29 14:04:01.323569 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:04:01.323527 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9\": container with ID starting with 20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9 not found: ID does not exist" containerID="20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9" Jul 29 14:04:01.323630 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.323577 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9"} err="failed to get container status \"20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9\": rpc error: code = NotFound desc = could not find container \"20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9\": container with ID starting with 20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9 not found: ID does not exist" Jul 29 14:04:01.323630 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.323601 2560 scope.go:117] "RemoveContainer" containerID="723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0" Jul 29 14:04:01.323901 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.323878 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0"} err="failed to get container status \"723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0\": rpc error: code = NotFound desc = could not find container \"723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0\": container with ID starting with 723c40ec423e64e3f9d2ab13d142c110a9d60f63263b8e50c60f7ec7bb94f1f0 not found: ID does not exist" Jul 29 14:04:01.323961 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.323903 2560 scope.go:117] "RemoveContainer" containerID="b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b" Jul 29 14:04:01.324143 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324125 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b"} err="failed to get container status \"b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b\": rpc error: code = NotFound desc = could not find container \"b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b\": container with ID starting with b273e32dd2d533c7a080f5ac40b1ee2e419a16d420cab9041bcf4b6f4df9e56b not found: ID does not exist" Jul 29 14:04:01.324212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324146 2560 scope.go:117] "RemoveContainer" containerID="78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3" Jul 29 14:04:01.324212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324164 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 14:04:01.324360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324340 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3"} err="failed to get container status \"78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3\": rpc error: code = NotFound desc = could not find container \"78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3\": container with ID starting with 78637c81ff37f63ab201f67fe74ff8d3676911974c2bf7a8940378ccbcee32d3 not found: ID does not exist" Jul 29 14:04:01.324422 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324362 2560 scope.go:117] "RemoveContainer" containerID="8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324515 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" containerName="registry" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324535 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" containerName="registry" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324564 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="config-reloader" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324573 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="config-reloader" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324582 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="prom-label-proxy" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324588 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="prom-label-proxy" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324597 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy-metric" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324601 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy-metric" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324598 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea"} err="failed to get container status \"8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea\": rpc error: code = NotFound desc = could not find container \"8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea\": container with ID starting with 8604ec28300cf192b07bf258be2c62b95a4af022f6f0d4e3f230c93565d578ea not found: ID does not exist" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324612 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="init-config-reloader" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324614 2560 scope.go:117] "RemoveContainer" containerID="d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324619 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="init-config-reloader" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324626 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="alertmanager" Jul 29 14:04:01.324532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324631 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="alertmanager" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324644 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy-web" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324649 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy-web" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324656 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324661 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324716 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e74d63-8e6d-4ccf-85a9-33c75ba7baa0" containerName="registry" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324728 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy-web" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324737 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="prom-label-proxy" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324749 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324759 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="kube-rbac-proxy-metric" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324766 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="alertmanager" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324772 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" containerName="config-reloader" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324867 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423"} err="failed to get container status \"d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423\": rpc error: code = NotFound desc = could not find container \"d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423\": container with ID starting with d6bf91270bdb86c87feca413735606f90a3518df467d0b4bd12c86294ae3e423 not found: ID does not exist" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.324891 2560 scope.go:117] "RemoveContainer" containerID="93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.325100 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7"} err="failed to get container status \"93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7\": rpc error: code = NotFound desc = could not find container \"93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7\": container with ID starting with 93625360f905a435ae7b0064b9635cacf4ac7db6267311ca2ea5db398f1e8db7 not found: ID does not exist" Jul 29 14:04:01.325137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.325121 2560 scope.go:117] "RemoveContainer" containerID="20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9" Jul 29 14:04:01.325562 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.325283 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9"} err="failed to get container status \"20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9\": rpc error: code = NotFound desc = could not find container \"20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9\": container with ID starting with 20fe590c3d2b210ad22923592d52093e0f7985cdf52055e2e63cdb87df68f4f9 not found: ID does not exist" Jul 29 14:04:01.329891 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.329875 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.332263 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.332244 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 14:04:01.332355 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.332330 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 14:04:01.332410 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.332330 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 14:04:01.332410 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.332377 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 14:04:01.332410 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.332387 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 14:04:01.332636 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.332610 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 14:04:01.332700 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.332685 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 14:04:01.332819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.332799 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wtqrm\"" Jul 29 14:04:01.332930 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.332841 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 14:04:01.337585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.337566 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 14:04:01.343739 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.343720 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 14:04:01.379164 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.379127 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b80753c-28d4-4885-bc7c-604b946d8b78" path="/var/lib/kubelet/pods/1b80753c-28d4-4885-bc7c-604b946d8b78/volumes" Jul 29 14:04:01.440789 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.440760 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-web-config\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.440992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.440793 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.440992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.440815 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.440992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.440833 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.440992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.440897 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.440992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.440917 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.440992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.440977 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xskxl\" (UniqueName: \"kubernetes.io/projected/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-kube-api-access-xskxl\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.440992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.440991 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.441289 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.441008 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.441289 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.441025 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.441289 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.441073 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-config-out\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.441289 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.441135 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.441289 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.441169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542449 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542361 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542449 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542407 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542449 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542430 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-web-config\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542449 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542450 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542470 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542514 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542530 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542580 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xskxl\" (UniqueName: \"kubernetes.io/projected/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-kube-api-access-xskxl\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542627 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542655 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.542784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542678 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-config-out\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.543254 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.542964 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.544197 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.543832 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.545858 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.545757 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.545969 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.545860 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-config-out\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.545969 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.545880 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.545969 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.545880 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.546124 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.546084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.546425 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.546398 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-web-config\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.546523 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.546473 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.546600 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.546573 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.546600 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.546576 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.548193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.548174 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.554241 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.554221 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xskxl\" (UniqueName: \"kubernetes.io/projected/7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c-kube-api-access-xskxl\") pod \"alertmanager-main-0\" (UID: \"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.640303 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.640274 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 14:04:01.775745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:01.775721 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 14:04:01.778353 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:04:01.778332 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7976e3de_69fd_4a4c_8c26_fc92fcbcaa9c.slice/crio-ef0882984a803c6f7d64b236788131194fd76c47a04e6cff6c5fbf41881fd189 WatchSource:0}: Error finding container ef0882984a803c6f7d64b236788131194fd76c47a04e6cff6c5fbf41881fd189: Status 404 returned error can't find the container with id ef0882984a803c6f7d64b236788131194fd76c47a04e6cff6c5fbf41881fd189 Jul 29 14:04:02.270417 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:02.270385 2560 generic.go:356] "Generic (PLEG): container finished" podID="7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c" containerID="2021a0990805e3e0eef02a3e9470bae4651ab6789c524cb796cbad33ad1c5baa" exitCode=0 Jul 29 14:04:02.270890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:02.270484 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c","Type":"ContainerDied","Data":"2021a0990805e3e0eef02a3e9470bae4651ab6789c524cb796cbad33ad1c5baa"} Jul 29 14:04:02.270890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:02.270620 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c","Type":"ContainerStarted","Data":"ef0882984a803c6f7d64b236788131194fd76c47a04e6cff6c5fbf41881fd189"} Jul 29 14:04:03.280074 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:03.280036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c","Type":"ContainerStarted","Data":"c5ecb0efbebb88e7a2aa91571e5dd3f320be3b5cb6a59fd08ca0d56234a0e051"} Jul 29 14:04:03.280587 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:03.280139 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c","Type":"ContainerStarted","Data":"5fae72890997a3f659c4090f85abb18429257e4bbf559b49611da6197c8b4514"} Jul 29 14:04:03.280587 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:03.280174 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c","Type":"ContainerStarted","Data":"96b3f5e8aa24029a091a12522406a73d8edf7fdec557406efff8cb0f223ce810"} Jul 29 14:04:03.280587 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:03.280202 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c","Type":"ContainerStarted","Data":"c75b70b9b303f6a1378af1682c45d5887599090b6e34e8ff8c8a872c1a5fa7cc"} Jul 29 14:04:03.280587 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:03.280224 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c","Type":"ContainerStarted","Data":"25270a4fb6f8f6d8a3c3ef9c0cae1999917ac3f35dee576fc2a862e05a1fd0c0"} Jul 29 14:04:03.280587 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:03.280247 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c","Type":"ContainerStarted","Data":"679a9fd7e717dc66255a64d99f764701cb4482c8b577860d168c7543d86152a4"} Jul 29 14:04:03.307599 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:04:03.307520 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.287400033 podStartE2EDuration="2.307500613s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.10058ms" isStatefulPod=false podCreationTimestamp="2026-07-29 14:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:04:03.306116252 +0000 UTC m=+176.507617425" watchObservedRunningTime="2026-07-29 14:04:03.307500613 +0000 UTC m=+176.509001812" Jul 29 14:05:11.846051 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:11.846013 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv"] Jul 29 14:05:11.849606 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:11.849589 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:11.852205 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:11.852186 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 14:05:11.853021 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:11.853004 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f5fpd\"" Jul 29 14:05:11.853074 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:11.853027 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 14:05:11.858864 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:11.858837 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv"] Jul 29 14:05:11.926213 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:11.926174 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:11.926213 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:11.926217 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:11.926429 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:11.926287 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrs4\" (UniqueName: \"kubernetes.io/projected/65feed3c-5add-41e1-b6bf-96dc3df286aa-kube-api-access-dwrs4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:12.027671 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:12.027627 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:12.027671 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:12.027678 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:12.027930 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:12.027708 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrs4\" (UniqueName: \"kubernetes.io/projected/65feed3c-5add-41e1-b6bf-96dc3df286aa-kube-api-access-dwrs4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:12.028095 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:12.028070 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:12.028153 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:12.028099 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:12.036818 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:12.036786 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrs4\" (UniqueName: \"kubernetes.io/projected/65feed3c-5add-41e1-b6bf-96dc3df286aa-kube-api-access-dwrs4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:12.159885 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:12.159785 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:12.299911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:12.299878 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv"] Jul 29 14:05:12.303940 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:05:12.303909 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65feed3c_5add_41e1_b6bf_96dc3df286aa.slice/crio-5965f95d33e1c6da4815fc72d5eea10beb54b0c5325c0e115efa34608c67f8e1 WatchSource:0}: Error finding container 5965f95d33e1c6da4815fc72d5eea10beb54b0c5325c0e115efa34608c67f8e1: Status 404 returned error can't find the container with id 5965f95d33e1c6da4815fc72d5eea10beb54b0c5325c0e115efa34608c67f8e1 Jul 29 14:05:12.545485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:12.545449 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" event={"ID":"65feed3c-5add-41e1-b6bf-96dc3df286aa","Type":"ContainerStarted","Data":"5965f95d33e1c6da4815fc72d5eea10beb54b0c5325c0e115efa34608c67f8e1"} Jul 29 14:05:19.577989 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:19.577902 2560 generic.go:356] "Generic (PLEG): container finished" podID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerID="2adac71f82cf8866d6f157b035386f75d954dd43567dd68f58050e5b1b564b8d" exitCode=0 Jul 29 14:05:19.577989 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:19.577950 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" event={"ID":"65feed3c-5add-41e1-b6bf-96dc3df286aa","Type":"ContainerDied","Data":"2adac71f82cf8866d6f157b035386f75d954dd43567dd68f58050e5b1b564b8d"} Jul 29 14:05:23.597118 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:23.597084 2560 generic.go:356] "Generic (PLEG): container finished" podID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerID="a6972a8a26960f522c54490374cfe02d2398e006b7aa68160e65038091a02d8e" exitCode=0 Jul 29 14:05:23.597504 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:23.597144 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" event={"ID":"65feed3c-5add-41e1-b6bf-96dc3df286aa","Type":"ContainerDied","Data":"a6972a8a26960f522c54490374cfe02d2398e006b7aa68160e65038091a02d8e"} Jul 29 14:05:32.642193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:32.642159 2560 generic.go:356] "Generic (PLEG): container finished" podID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerID="7322926e3bf5445479bfe7c8ea44ab772f5812d8c550e1e76b0a1e4039f07e43" exitCode=0 Jul 29 14:05:32.642606 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:32.642221 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" event={"ID":"65feed3c-5add-41e1-b6bf-96dc3df286aa","Type":"ContainerDied","Data":"7322926e3bf5445479bfe7c8ea44ab772f5812d8c550e1e76b0a1e4039f07e43"} Jul 29 14:05:33.780265 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.780231 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:33.820113 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.820089 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-util\") pod \"65feed3c-5add-41e1-b6bf-96dc3df286aa\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " Jul 29 14:05:33.820278 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.820119 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-bundle\") pod \"65feed3c-5add-41e1-b6bf-96dc3df286aa\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " Jul 29 14:05:33.820278 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.820158 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwrs4\" (UniqueName: \"kubernetes.io/projected/65feed3c-5add-41e1-b6bf-96dc3df286aa-kube-api-access-dwrs4\") pod \"65feed3c-5add-41e1-b6bf-96dc3df286aa\" (UID: \"65feed3c-5add-41e1-b6bf-96dc3df286aa\") " Jul 29 14:05:33.820891 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.820862 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-bundle" (OuterVolumeSpecName: "bundle") pod "65feed3c-5add-41e1-b6bf-96dc3df286aa" (UID: "65feed3c-5add-41e1-b6bf-96dc3df286aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:05:33.822504 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.822479 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65feed3c-5add-41e1-b6bf-96dc3df286aa-kube-api-access-dwrs4" (OuterVolumeSpecName: "kube-api-access-dwrs4") pod "65feed3c-5add-41e1-b6bf-96dc3df286aa" (UID: "65feed3c-5add-41e1-b6bf-96dc3df286aa"). InnerVolumeSpecName "kube-api-access-dwrs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:05:33.826000 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.825975 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-util" (OuterVolumeSpecName: "util") pod "65feed3c-5add-41e1-b6bf-96dc3df286aa" (UID: "65feed3c-5add-41e1-b6bf-96dc3df286aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:05:33.921596 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.921490 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-util\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:05:33.921596 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.921518 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65feed3c-5add-41e1-b6bf-96dc3df286aa-bundle\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:05:33.921596 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:33.921534 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwrs4\" (UniqueName: \"kubernetes.io/projected/65feed3c-5add-41e1-b6bf-96dc3df286aa-kube-api-access-dwrs4\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:05:34.651485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:34.651451 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" event={"ID":"65feed3c-5add-41e1-b6bf-96dc3df286aa","Type":"ContainerDied","Data":"5965f95d33e1c6da4815fc72d5eea10beb54b0c5325c0e115efa34608c67f8e1"} Jul 29 14:05:34.651677 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:34.651499 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5965f95d33e1c6da4815fc72d5eea10beb54b0c5325c0e115efa34608c67f8e1" Jul 29 14:05:34.651677 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:34.651498 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540t79sv" Jul 29 14:05:45.435773 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.435740 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s"] Jul 29 14:05:45.436216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.436051 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerName="pull" Jul 29 14:05:45.436216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.436063 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerName="pull" Jul 29 14:05:45.436216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.436078 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerName="util" Jul 29 14:05:45.436216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.436083 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerName="util" Jul 29 14:05:45.436216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.436089 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerName="extract" Jul 29 14:05:45.436216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.436094 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerName="extract" Jul 29 14:05:45.436216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.436178 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="65feed3c-5add-41e1-b6bf-96dc3df286aa" containerName="extract" Jul 29 14:05:45.439471 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.439454 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.442322 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.442304 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 14:05:45.442459 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.442442 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 14:05:45.443178 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.443162 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f5fpd\"" Jul 29 14:05:45.455028 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.455005 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s"] Jul 29 14:05:45.515085 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.515048 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzbf\" (UniqueName: \"kubernetes.io/projected/6d863641-7c36-4c8e-91d6-635329735075-kube-api-access-xjzbf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.515244 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.515148 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.515244 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.515179 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.616060 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.616027 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.616060 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.616060 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.616311 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.616153 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzbf\" (UniqueName: \"kubernetes.io/projected/6d863641-7c36-4c8e-91d6-635329735075-kube-api-access-xjzbf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.616380 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.616366 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.616455 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.616435 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.626126 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.626104 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzbf\" (UniqueName: \"kubernetes.io/projected/6d863641-7c36-4c8e-91d6-635329735075-kube-api-access-xjzbf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.748847 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.748759 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:45.887303 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:45.887268 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s"] Jul 29 14:05:45.890412 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:05:45.890383 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d863641_7c36_4c8e_91d6_635329735075.slice/crio-fd1c1134adc4d75f5740703d0ade543eb4c9edf2a29b30df721e4299a8657340 WatchSource:0}: Error finding container fd1c1134adc4d75f5740703d0ade543eb4c9edf2a29b30df721e4299a8657340: Status 404 returned error can't find the container with id fd1c1134adc4d75f5740703d0ade543eb4c9edf2a29b30df721e4299a8657340 Jul 29 14:05:46.700987 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:46.700950 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d863641-7c36-4c8e-91d6-635329735075" containerID="bf3ae47d36f3e9689212ca432a81e2f16148192704af1d4588e01c1cb28dafff" exitCode=0 Jul 29 14:05:46.701343 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:46.701036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" event={"ID":"6d863641-7c36-4c8e-91d6-635329735075","Type":"ContainerDied","Data":"bf3ae47d36f3e9689212ca432a81e2f16148192704af1d4588e01c1cb28dafff"} Jul 29 14:05:46.701343 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:46.701079 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" event={"ID":"6d863641-7c36-4c8e-91d6-635329735075","Type":"ContainerStarted","Data":"fd1c1134adc4d75f5740703d0ade543eb4c9edf2a29b30df721e4299a8657340"} Jul 29 14:05:46.979158 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:46.979129 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2"] Jul 29 14:05:46.981135 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:46.981121 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" Jul 29 14:05:46.983438 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:46.983414 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 14:05:46.983574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:46.983465 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lmhgb\"" Jul 29 14:05:46.983574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:46.983515 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 14:05:46.991646 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:46.991625 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2"] Jul 29 14:05:47.027693 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:47.027663 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4gdl\" (UniqueName: \"kubernetes.io/projected/e73085a5-9bd3-46d9-b4c5-907ea0b73a68-kube-api-access-k4gdl\") pod \"cert-manager-cainjector-559cdd8c87-jk7s2\" (UID: \"e73085a5-9bd3-46d9-b4c5-907ea0b73a68\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" Jul 29 14:05:47.027809 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:47.027703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e73085a5-9bd3-46d9-b4c5-907ea0b73a68-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jk7s2\" (UID: \"e73085a5-9bd3-46d9-b4c5-907ea0b73a68\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" Jul 29 14:05:47.128168 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:47.128138 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4gdl\" (UniqueName: \"kubernetes.io/projected/e73085a5-9bd3-46d9-b4c5-907ea0b73a68-kube-api-access-k4gdl\") pod \"cert-manager-cainjector-559cdd8c87-jk7s2\" (UID: \"e73085a5-9bd3-46d9-b4c5-907ea0b73a68\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" Jul 29 14:05:47.128342 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:47.128183 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e73085a5-9bd3-46d9-b4c5-907ea0b73a68-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jk7s2\" (UID: \"e73085a5-9bd3-46d9-b4c5-907ea0b73a68\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" Jul 29 14:05:47.139244 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:47.139218 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e73085a5-9bd3-46d9-b4c5-907ea0b73a68-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jk7s2\" (UID: \"e73085a5-9bd3-46d9-b4c5-907ea0b73a68\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" Jul 29 14:05:47.139395 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:47.139371 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4gdl\" (UniqueName: \"kubernetes.io/projected/e73085a5-9bd3-46d9-b4c5-907ea0b73a68-kube-api-access-k4gdl\") pod \"cert-manager-cainjector-559cdd8c87-jk7s2\" (UID: \"e73085a5-9bd3-46d9-b4c5-907ea0b73a68\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" Jul 29 14:05:47.305210 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:47.305129 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" Jul 29 14:05:47.434491 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:47.434468 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2"] Jul 29 14:05:47.437014 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:05:47.436987 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73085a5_9bd3_46d9_b4c5_907ea0b73a68.slice/crio-778190e4733d3fa931dd35651eb9500bddb280d66d99ea8a76f817b89613ce92 WatchSource:0}: Error finding container 778190e4733d3fa931dd35651eb9500bddb280d66d99ea8a76f817b89613ce92: Status 404 returned error can't find the container with id 778190e4733d3fa931dd35651eb9500bddb280d66d99ea8a76f817b89613ce92 Jul 29 14:05:47.706076 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:47.706049 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" event={"ID":"e73085a5-9bd3-46d9-b4c5-907ea0b73a68","Type":"ContainerStarted","Data":"778190e4733d3fa931dd35651eb9500bddb280d66d99ea8a76f817b89613ce92"} Jul 29 14:05:50.721458 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:50.721420 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d863641-7c36-4c8e-91d6-635329735075" containerID="ba1d689fa00488cef36559c7dbbcf6b909655d814650b4b6dba7eef1dd40e640" exitCode=0 Jul 29 14:05:50.722038 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:50.721503 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" event={"ID":"6d863641-7c36-4c8e-91d6-635329735075","Type":"ContainerDied","Data":"ba1d689fa00488cef36559c7dbbcf6b909655d814650b4b6dba7eef1dd40e640"} Jul 29 14:05:51.726614 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:51.726580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" event={"ID":"e73085a5-9bd3-46d9-b4c5-907ea0b73a68","Type":"ContainerStarted","Data":"db305254c996163ae49ae531488b9f3da7a5a27f5d3b2f374b4a7f01760cd7d0"} Jul 29 14:05:51.728500 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:51.728474 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d863641-7c36-4c8e-91d6-635329735075" containerID="9ff97c1d935461df73c0c71e6b914238fc26214c2bcbdfb6e1f280518cb0cb54" exitCode=0 Jul 29 14:05:51.728632 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:51.728568 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" event={"ID":"6d863641-7c36-4c8e-91d6-635329735075","Type":"ContainerDied","Data":"9ff97c1d935461df73c0c71e6b914238fc26214c2bcbdfb6e1f280518cb0cb54"} Jul 29 14:05:51.743863 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:51.743819 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jk7s2" podStartSLOduration=2.195431429 podStartE2EDuration="5.743805427s" totalImagesPullingTime="3.548373998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:05:51.742238082 +0000 UTC m=+284.943739253" watchObservedRunningTime="2026-07-29 14:05:51.743805427 +0000 UTC m=+284.945306598" Jul 29 14:05:52.864226 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:52.864202 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:52.982133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:52.982096 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjzbf\" (UniqueName: \"kubernetes.io/projected/6d863641-7c36-4c8e-91d6-635329735075-kube-api-access-xjzbf\") pod \"6d863641-7c36-4c8e-91d6-635329735075\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " Jul 29 14:05:52.982282 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:52.982223 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-bundle\") pod \"6d863641-7c36-4c8e-91d6-635329735075\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " Jul 29 14:05:52.982282 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:52.982248 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-util\") pod \"6d863641-7c36-4c8e-91d6-635329735075\" (UID: \"6d863641-7c36-4c8e-91d6-635329735075\") " Jul 29 14:05:52.982621 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:52.982597 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-bundle" (OuterVolumeSpecName: "bundle") pod "6d863641-7c36-4c8e-91d6-635329735075" (UID: "6d863641-7c36-4c8e-91d6-635329735075"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:05:52.984291 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:52.984239 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d863641-7c36-4c8e-91d6-635329735075-kube-api-access-xjzbf" (OuterVolumeSpecName: "kube-api-access-xjzbf") pod "6d863641-7c36-4c8e-91d6-635329735075" (UID: "6d863641-7c36-4c8e-91d6-635329735075"). InnerVolumeSpecName "kube-api-access-xjzbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:05:52.986633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:52.986601 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-util" (OuterVolumeSpecName: "util") pod "6d863641-7c36-4c8e-91d6-635329735075" (UID: "6d863641-7c36-4c8e-91d6-635329735075"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:05:53.083520 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:53.083486 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-bundle\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:05:53.083520 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:53.083512 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d863641-7c36-4c8e-91d6-635329735075-util\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:05:53.083520 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:53.083522 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjzbf\" (UniqueName: \"kubernetes.io/projected/6d863641-7c36-4c8e-91d6-635329735075-kube-api-access-xjzbf\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:05:53.738487 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:53.738451 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" event={"ID":"6d863641-7c36-4c8e-91d6-635329735075","Type":"ContainerDied","Data":"fd1c1134adc4d75f5740703d0ade543eb4c9edf2a29b30df721e4299a8657340"} Jul 29 14:05:53.738675 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:53.738512 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd1c1134adc4d75f5740703d0ade543eb4c9edf2a29b30df721e4299a8657340" Jul 29 14:05:53.738675 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:53.738468 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjqf6s" Jul 29 14:05:58.579130 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.579093 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl"] Jul 29 14:05:58.579503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.579400 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d863641-7c36-4c8e-91d6-635329735075" containerName="util" Jul 29 14:05:58.579503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.579410 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d863641-7c36-4c8e-91d6-635329735075" containerName="util" Jul 29 14:05:58.579503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.579428 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d863641-7c36-4c8e-91d6-635329735075" containerName="pull" Jul 29 14:05:58.579503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.579434 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d863641-7c36-4c8e-91d6-635329735075" containerName="pull" Jul 29 14:05:58.579503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.579440 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d863641-7c36-4c8e-91d6-635329735075" containerName="extract" Jul 29 14:05:58.579503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.579445 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d863641-7c36-4c8e-91d6-635329735075" containerName="extract" Jul 29 14:05:58.579503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.579485 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d863641-7c36-4c8e-91d6-635329735075" containerName="extract" Jul 29 14:05:58.599101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.599075 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl"] Jul 29 14:05:58.599227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.599161 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" Jul 29 14:05:58.601796 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.601768 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 14:05:58.602559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.602515 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-mgqdf\"" Jul 29 14:05:58.602677 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.602518 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 14:05:58.729118 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.729074 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/275e0fbc-1694-4401-b84f-bf64cbe6eb78-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lwjpl\" (UID: \"275e0fbc-1694-4401-b84f-bf64cbe6eb78\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" Jul 29 14:05:58.729296 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.729208 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhln2\" (UniqueName: \"kubernetes.io/projected/275e0fbc-1694-4401-b84f-bf64cbe6eb78-kube-api-access-xhln2\") pod \"openshift-lws-operator-fd8ccff4c-lwjpl\" (UID: \"275e0fbc-1694-4401-b84f-bf64cbe6eb78\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" Jul 29 14:05:58.830282 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.830203 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhln2\" (UniqueName: \"kubernetes.io/projected/275e0fbc-1694-4401-b84f-bf64cbe6eb78-kube-api-access-xhln2\") pod \"openshift-lws-operator-fd8ccff4c-lwjpl\" (UID: \"275e0fbc-1694-4401-b84f-bf64cbe6eb78\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" Jul 29 14:05:58.830282 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.830249 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/275e0fbc-1694-4401-b84f-bf64cbe6eb78-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lwjpl\" (UID: \"275e0fbc-1694-4401-b84f-bf64cbe6eb78\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" Jul 29 14:05:58.830636 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.830619 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/275e0fbc-1694-4401-b84f-bf64cbe6eb78-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lwjpl\" (UID: \"275e0fbc-1694-4401-b84f-bf64cbe6eb78\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" Jul 29 14:05:58.840808 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.840775 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhln2\" (UniqueName: \"kubernetes.io/projected/275e0fbc-1694-4401-b84f-bf64cbe6eb78-kube-api-access-xhln2\") pod \"openshift-lws-operator-fd8ccff4c-lwjpl\" (UID: \"275e0fbc-1694-4401-b84f-bf64cbe6eb78\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" Jul 29 14:05:58.908430 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:58.908398 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" Jul 29 14:05:59.042856 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:59.042826 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl"] Jul 29 14:05:59.045464 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:05:59.045439 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod275e0fbc_1694_4401_b84f_bf64cbe6eb78.slice/crio-9223ab1301f038440a202ee679a4146e1c18b67df5687a9a08909128c2228914 WatchSource:0}: Error finding container 9223ab1301f038440a202ee679a4146e1c18b67df5687a9a08909128c2228914: Status 404 returned error can't find the container with id 9223ab1301f038440a202ee679a4146e1c18b67df5687a9a08909128c2228914 Jul 29 14:05:59.767124 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:05:59.767090 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" event={"ID":"275e0fbc-1694-4401-b84f-bf64cbe6eb78","Type":"ContainerStarted","Data":"9223ab1301f038440a202ee679a4146e1c18b67df5687a9a08909128c2228914"} Jul 29 14:06:01.777634 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:01.777590 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" event={"ID":"275e0fbc-1694-4401-b84f-bf64cbe6eb78","Type":"ContainerStarted","Data":"573b2ad5680b9e211044039b6a10360539831bf53376c5488182337139a11a26"} Jul 29 14:06:01.795134 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:01.795074 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwjpl" podStartSLOduration=1.261894381 podStartE2EDuration="3.795052739s" totalImagesPullingTime="2.533158358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:06:01.793487202 +0000 UTC m=+294.994988376" watchObservedRunningTime="2026-07-29 14:06:01.795052739 +0000 UTC m=+294.996553913" Jul 29 14:06:04.332495 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.332459 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf"] Jul 29 14:06:04.335166 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.335150 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.337596 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.337570 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 14:06:04.337732 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.337717 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f5fpd\"" Jul 29 14:06:04.338446 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.338432 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 14:06:04.347849 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.347821 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf"] Jul 29 14:06:04.480649 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.480602 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.480841 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.480668 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.480939 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.480837 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqfk4\" (UniqueName: \"kubernetes.io/projected/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-kube-api-access-zqfk4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.581363 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.581317 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.581586 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.581377 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqfk4\" (UniqueName: \"kubernetes.io/projected/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-kube-api-access-zqfk4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.581586 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.581456 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.581735 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.581678 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.581840 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.581822 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.597865 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.597797 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqfk4\" (UniqueName: \"kubernetes.io/projected/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-kube-api-access-zqfk4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.644345 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.644285 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:04.794928 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:04.794903 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf"] Jul 29 14:06:04.797002 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:06:04.796976 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bb9f90_3da0_4ec0_b653_d7e20b5cf8f9.slice/crio-09854e94001453da84891e52e0d20524188b5a574d16827eb43f33967480dbca WatchSource:0}: Error finding container 09854e94001453da84891e52e0d20524188b5a574d16827eb43f33967480dbca: Status 404 returned error can't find the container with id 09854e94001453da84891e52e0d20524188b5a574d16827eb43f33967480dbca Jul 29 14:06:05.796803 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:05.796774 2560 generic.go:356] "Generic (PLEG): container finished" podID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerID="d9afda909f464f981a58dfe3b5654e5d7493534f2325b94ffa2ad4c4a13cc093" exitCode=0 Jul 29 14:06:05.797181 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:05.796830 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" event={"ID":"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9","Type":"ContainerDied","Data":"d9afda909f464f981a58dfe3b5654e5d7493534f2325b94ffa2ad4c4a13cc093"} Jul 29 14:06:05.797181 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:05.796865 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" event={"ID":"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9","Type":"ContainerStarted","Data":"09854e94001453da84891e52e0d20524188b5a574d16827eb43f33967480dbca"} Jul 29 14:06:07.286046 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:07.286014 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:06:07.288885 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:07.288862 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:06:07.296033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:07.296000 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:06:07.299525 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:07.299500 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:06:07.304431 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:07.304403 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 14:06:07.808970 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:07.808873 2560 generic.go:356] "Generic (PLEG): container finished" podID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerID="e0f8b5af659fb9e784527b0553d97b11c783a3a350dd1b46d9c4b854cae7fdf5" exitCode=0 Jul 29 14:06:07.808970 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:07.808916 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" event={"ID":"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9","Type":"ContainerDied","Data":"e0f8b5af659fb9e784527b0553d97b11c783a3a350dd1b46d9c4b854cae7fdf5"} Jul 29 14:06:08.816015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:08.815986 2560 generic.go:356] "Generic (PLEG): container finished" podID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerID="882fbe34a007934a5a58db93f1f201c6ad1698725451a77c9ec4cce4b976e651" exitCode=0 Jul 29 14:06:08.816376 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:08.816075 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" event={"ID":"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9","Type":"ContainerDied","Data":"882fbe34a007934a5a58db93f1f201c6ad1698725451a77c9ec4cce4b976e651"} Jul 29 14:06:09.946829 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:09.946803 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:10.026350 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.026325 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-util\") pod \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " Jul 29 14:06:10.026512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.026361 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqfk4\" (UniqueName: \"kubernetes.io/projected/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-kube-api-access-zqfk4\") pod \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " Jul 29 14:06:10.026512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.026399 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-bundle\") pod \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\" (UID: \"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9\") " Jul 29 14:06:10.027191 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.027150 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-bundle" (OuterVolumeSpecName: "bundle") pod "d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" (UID: "d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:06:10.028571 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.028530 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-kube-api-access-zqfk4" (OuterVolumeSpecName: "kube-api-access-zqfk4") pod "d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" (UID: "d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9"). InnerVolumeSpecName "kube-api-access-zqfk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:06:10.037470 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.037442 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-util" (OuterVolumeSpecName: "util") pod "d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" (UID: "d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:06:10.127856 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.127792 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-util\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:10.127856 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.127812 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqfk4\" (UniqueName: \"kubernetes.io/projected/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-kube-api-access-zqfk4\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:10.127856 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.127823 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9-bundle\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:10.826406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.826370 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" event={"ID":"d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9","Type":"ContainerDied","Data":"09854e94001453da84891e52e0d20524188b5a574d16827eb43f33967480dbca"} Jul 29 14:06:10.826406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.826396 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs7nsf" Jul 29 14:06:10.826641 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:10.826418 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09854e94001453da84891e52e0d20524188b5a574d16827eb43f33967480dbca" Jul 29 14:06:21.594439 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.594401 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9"] Jul 29 14:06:21.594889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.594737 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerName="pull" Jul 29 14:06:21.594889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.594748 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerName="pull" Jul 29 14:06:21.594889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.594760 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerName="extract" Jul 29 14:06:21.594889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.594765 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerName="extract" Jul 29 14:06:21.594889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.594772 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerName="util" Jul 29 14:06:21.594889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.594779 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerName="util" Jul 29 14:06:21.594889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.594837 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1bb9f90-3da0-4ec0-b653-d7e20b5cf8f9" containerName="extract" Jul 29 14:06:21.596869 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.596852 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.600082 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.600064 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 14:06:21.600190 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.600109 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 14:06:21.600190 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.600163 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 14:06:21.600523 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.600505 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 14:06:21.600634 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.600585 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2g49j\"" Jul 29 14:06:21.618122 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.618097 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9"] Jul 29 14:06:21.723710 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.723678 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-manifests\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.723894 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.723728 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.723894 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.723802 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbpc\" (UniqueName: \"kubernetes.io/projected/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-kube-api-access-xlbpc\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.723894 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.723841 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-tmp\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.724028 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.723917 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.825134 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.825100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.825318 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.825148 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-manifests\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.825318 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.825191 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.825318 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.825229 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbpc\" (UniqueName: \"kubernetes.io/projected/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-kube-api-access-xlbpc\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.825478 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.825363 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-tmp\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.825680 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.825655 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-manifests\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.827594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.827568 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-tmp\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.827705 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.827646 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.827769 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.827706 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.833568 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.833521 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbpc\" (UniqueName: \"kubernetes.io/projected/79a4777a-ea6f-4969-ab7e-5a18b5c253cd-kube-api-access-xlbpc\") pod \"opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9\" (UID: \"79a4777a-ea6f-4969-ab7e-5a18b5c253cd\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:21.907231 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:21.907152 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:22.046403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:22.046333 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9"] Jul 29 14:06:22.052102 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:06:22.052066 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a4777a_ea6f_4969_ab7e_5a18b5c253cd.slice/crio-e6883cde1c7b9cbee7aaa77869d5962c5c6018bb1aa319fc1017b1bd9a1607d4 WatchSource:0}: Error finding container e6883cde1c7b9cbee7aaa77869d5962c5c6018bb1aa319fc1017b1bd9a1607d4: Status 404 returned error can't find the container with id e6883cde1c7b9cbee7aaa77869d5962c5c6018bb1aa319fc1017b1bd9a1607d4 Jul 29 14:06:22.053484 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:22.053464 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:06:22.882880 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:22.882842 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" event={"ID":"79a4777a-ea6f-4969-ab7e-5a18b5c253cd","Type":"ContainerStarted","Data":"e6883cde1c7b9cbee7aaa77869d5962c5c6018bb1aa319fc1017b1bd9a1607d4"} Jul 29 14:06:25.901743 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:25.901706 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" event={"ID":"79a4777a-ea6f-4969-ab7e-5a18b5c253cd","Type":"ContainerStarted","Data":"34a88167d0a5dd133ac540d20f1a44e7c3896342f742320117ab8e060d9a02e5"} Jul 29 14:06:26.908011 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:26.907974 2560 generic.go:356] "Generic (PLEG): container finished" podID="79a4777a-ea6f-4969-ab7e-5a18b5c253cd" containerID="34a88167d0a5dd133ac540d20f1a44e7c3896342f742320117ab8e060d9a02e5" exitCode=0 Jul 29 14:06:26.908400 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:26.908065 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" event={"ID":"79a4777a-ea6f-4969-ab7e-5a18b5c253cd","Type":"ContainerDied","Data":"34a88167d0a5dd133ac540d20f1a44e7c3896342f742320117ab8e060d9a02e5"} Jul 29 14:06:27.915779 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:27.915732 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" event={"ID":"79a4777a-ea6f-4969-ab7e-5a18b5c253cd","Type":"ContainerStarted","Data":"7f01faac3345d276632eea56d0878554ef738bfd0708e338c802cc27961a67ad"} Jul 29 14:06:27.916266 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:27.915868 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:27.940893 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:27.940828 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" podStartSLOduration=2.657877567 podStartE2EDuration="6.940813211s" totalImagesPullingTime="3.25504324s" totalInitContainerRuntime="1.027892404s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:06:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:06:27.93824934 +0000 UTC m=+321.139750532" watchObservedRunningTime="2026-07-29 14:06:27.940813211 +0000 UTC m=+321.142314383" Jul 29 14:06:38.923279 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:38.923243 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9" Jul 29 14:06:41.658241 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.658209 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv"] Jul 29 14:06:41.663095 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.663072 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.669933 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.669706 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 14:06:41.669933 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.669798 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 14:06:41.684741 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.684710 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv"] Jul 29 14:06:41.687853 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.687827 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20eadaef-47b1-46fb-b47d-d9ca8d1f8931-tls-certs\") pod \"kube-auth-proxy-6b4bcbd496-v65rv\" (UID: \"20eadaef-47b1-46fb-b47d-d9ca8d1f8931\") " pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.687992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.687897 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20eadaef-47b1-46fb-b47d-d9ca8d1f8931-tmp\") pod \"kube-auth-proxy-6b4bcbd496-v65rv\" (UID: \"20eadaef-47b1-46fb-b47d-d9ca8d1f8931\") " pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.687992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.687937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrr4\" (UniqueName: \"kubernetes.io/projected/20eadaef-47b1-46fb-b47d-d9ca8d1f8931-kube-api-access-hxrr4\") pod \"kube-auth-proxy-6b4bcbd496-v65rv\" (UID: \"20eadaef-47b1-46fb-b47d-d9ca8d1f8931\") " pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.789443 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.789400 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20eadaef-47b1-46fb-b47d-d9ca8d1f8931-tmp\") pod \"kube-auth-proxy-6b4bcbd496-v65rv\" (UID: \"20eadaef-47b1-46fb-b47d-d9ca8d1f8931\") " pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.789443 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.789450 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrr4\" (UniqueName: \"kubernetes.io/projected/20eadaef-47b1-46fb-b47d-d9ca8d1f8931-kube-api-access-hxrr4\") pod \"kube-auth-proxy-6b4bcbd496-v65rv\" (UID: \"20eadaef-47b1-46fb-b47d-d9ca8d1f8931\") " pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.789719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.789529 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20eadaef-47b1-46fb-b47d-d9ca8d1f8931-tls-certs\") pod \"kube-auth-proxy-6b4bcbd496-v65rv\" (UID: \"20eadaef-47b1-46fb-b47d-d9ca8d1f8931\") " pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.792038 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.791999 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20eadaef-47b1-46fb-b47d-d9ca8d1f8931-tmp\") pod \"kube-auth-proxy-6b4bcbd496-v65rv\" (UID: \"20eadaef-47b1-46fb-b47d-d9ca8d1f8931\") " pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.792183 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.792155 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20eadaef-47b1-46fb-b47d-d9ca8d1f8931-tls-certs\") pod \"kube-auth-proxy-6b4bcbd496-v65rv\" (UID: \"20eadaef-47b1-46fb-b47d-d9ca8d1f8931\") " pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.807068 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.807044 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrr4\" (UniqueName: \"kubernetes.io/projected/20eadaef-47b1-46fb-b47d-d9ca8d1f8931-kube-api-access-hxrr4\") pod \"kube-auth-proxy-6b4bcbd496-v65rv\" (UID: \"20eadaef-47b1-46fb-b47d-d9ca8d1f8931\") " pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:41.972718 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:41.972687 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" Jul 29 14:06:42.086616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.086582 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mhpqh"] Jul 29 14:06:42.090387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.090369 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:42.093274 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.093253 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 14:06:42.093623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.093605 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pnfnz\"" Jul 29 14:06:42.102105 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.102075 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mhpqh"] Jul 29 14:06:42.122737 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:06:42.122707 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20eadaef_47b1_46fb_b47d_d9ca8d1f8931.slice/crio-2604302baa0b9d9ae24fe5f6e124fdd2b63140abb22e81f65c1c612d51106fce WatchSource:0}: Error finding container 2604302baa0b9d9ae24fe5f6e124fdd2b63140abb22e81f65c1c612d51106fce: Status 404 returned error can't find the container with id 2604302baa0b9d9ae24fe5f6e124fdd2b63140abb22e81f65c1c612d51106fce Jul 29 14:06:42.123918 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.123887 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv"] Jul 29 14:06:42.193379 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.193346 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77b01920-4ef0-42f1-b160-c11a12fc72c8-cert\") pod \"odh-model-controller-7998c67dc4-mhpqh\" (UID: \"77b01920-4ef0-42f1-b160-c11a12fc72c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:42.193548 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.193407 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5s99\" (UniqueName: \"kubernetes.io/projected/77b01920-4ef0-42f1-b160-c11a12fc72c8-kube-api-access-r5s99\") pod \"odh-model-controller-7998c67dc4-mhpqh\" (UID: \"77b01920-4ef0-42f1-b160-c11a12fc72c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:42.294645 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.294527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77b01920-4ef0-42f1-b160-c11a12fc72c8-cert\") pod \"odh-model-controller-7998c67dc4-mhpqh\" (UID: \"77b01920-4ef0-42f1-b160-c11a12fc72c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:42.294645 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.294579 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5s99\" (UniqueName: \"kubernetes.io/projected/77b01920-4ef0-42f1-b160-c11a12fc72c8-kube-api-access-r5s99\") pod \"odh-model-controller-7998c67dc4-mhpqh\" (UID: \"77b01920-4ef0-42f1-b160-c11a12fc72c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:42.294871 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:06:42.294699 2560 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 29 14:06:42.294871 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:06:42.294776 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77b01920-4ef0-42f1-b160-c11a12fc72c8-cert podName:77b01920-4ef0-42f1-b160-c11a12fc72c8 nodeName:}" failed. No retries permitted until 2026-07-29 14:06:42.794753589 +0000 UTC m=+335.996254743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/77b01920-4ef0-42f1-b160-c11a12fc72c8-cert") pod "odh-model-controller-7998c67dc4-mhpqh" (UID: "77b01920-4ef0-42f1-b160-c11a12fc72c8") : secret "odh-model-controller-webhook-cert" not found Jul 29 14:06:42.319111 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.319078 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5s99\" (UniqueName: \"kubernetes.io/projected/77b01920-4ef0-42f1-b160-c11a12fc72c8-kube-api-access-r5s99\") pod \"odh-model-controller-7998c67dc4-mhpqh\" (UID: \"77b01920-4ef0-42f1-b160-c11a12fc72c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:42.798595 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.798529 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77b01920-4ef0-42f1-b160-c11a12fc72c8-cert\") pod \"odh-model-controller-7998c67dc4-mhpqh\" (UID: \"77b01920-4ef0-42f1-b160-c11a12fc72c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:42.801486 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.801461 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77b01920-4ef0-42f1-b160-c11a12fc72c8-cert\") pod \"odh-model-controller-7998c67dc4-mhpqh\" (UID: \"77b01920-4ef0-42f1-b160-c11a12fc72c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:42.985688 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:42.985636 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" event={"ID":"20eadaef-47b1-46fb-b47d-d9ca8d1f8931","Type":"ContainerStarted","Data":"2604302baa0b9d9ae24fe5f6e124fdd2b63140abb22e81f65c1c612d51106fce"} Jul 29 14:06:43.003335 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.003293 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:43.147532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.147399 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck"] Jul 29 14:06:43.155131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.154706 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.157936 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.157722 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 14:06:43.157936 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.157729 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5hpqr\"" Jul 29 14:06:43.158148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.157990 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 14:06:43.158148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.158024 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 14:06:43.158263 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.158179 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 14:06:43.176255 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.176194 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck"] Jul 29 14:06:43.200099 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.200020 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mhpqh"] Jul 29 14:06:43.203257 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:06:43.202638 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b01920_4ef0_42f1_b160_c11a12fc72c8.slice/crio-66d3575ecf4173008ef6a89ad253082f6e00e4caa3e268933ff44242066bc7c8 WatchSource:0}: Error finding container 66d3575ecf4173008ef6a89ad253082f6e00e4caa3e268933ff44242066bc7c8: Status 404 returned error can't find the container with id 66d3575ecf4173008ef6a89ad253082f6e00e4caa3e268933ff44242066bc7c8 Jul 29 14:06:43.209035 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.209006 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.209156 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.209096 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33461ba-7347-4d89-91dc-e4d94c280cd5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.209156 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.209150 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.209364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.209180 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.209364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.209219 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.209364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.209244 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.209364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.209271 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4f9c\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-kube-api-access-p4f9c\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.310333 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.310298 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33461ba-7347-4d89-91dc-e4d94c280cd5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.310733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.310357 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.310733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.310468 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.310733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.310523 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.310733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.310570 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.310733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.310599 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f9c\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-kube-api-access-p4f9c\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.310733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.310672 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.311373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.311343 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.314101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.314053 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.314684 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.314385 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33461ba-7347-4d89-91dc-e4d94c280cd5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.314684 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.314651 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.316173 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.316150 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.325307 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.325242 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.326200 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.326178 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4f9c\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-kube-api-access-p4f9c\") pod \"istiod-openshift-gateway-7b7995d5cd-5rjck\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.468619 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.468581 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:43.626395 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.626308 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck"] Jul 29 14:06:43.630763 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:06:43.630721 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33461ba_7347_4d89_91dc_e4d94c280cd5.slice/crio-6a55129b2799ba9333e2076b1237a467d43642d16abe537e724c97127ba2d57d WatchSource:0}: Error finding container 6a55129b2799ba9333e2076b1237a467d43642d16abe537e724c97127ba2d57d: Status 404 returned error can't find the container with id 6a55129b2799ba9333e2076b1237a467d43642d16abe537e724c97127ba2d57d Jul 29 14:06:43.993748 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.993711 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" event={"ID":"e33461ba-7347-4d89-91dc-e4d94c280cd5","Type":"ContainerStarted","Data":"6a55129b2799ba9333e2076b1237a467d43642d16abe537e724c97127ba2d57d"} Jul 29 14:06:43.995075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:43.995051 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" event={"ID":"77b01920-4ef0-42f1-b160-c11a12fc72c8","Type":"ContainerStarted","Data":"66d3575ecf4173008ef6a89ad253082f6e00e4caa3e268933ff44242066bc7c8"} Jul 29 14:06:45.006101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:45.005952 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" event={"ID":"20eadaef-47b1-46fb-b47d-d9ca8d1f8931","Type":"ContainerStarted","Data":"d79abcfc9ef1a726af152bc0737550c45a0427e08e56185325266daabdefcae7"} Jul 29 14:06:45.025787 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:45.024606 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b4bcbd496-v65rv" podStartSLOduration=1.280131844 podStartE2EDuration="4.024585644s" totalImagesPullingTime="2.7444538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:06:45.022907283 +0000 UTC m=+338.224408457" watchObservedRunningTime="2026-07-29 14:06:45.024585644 +0000 UTC m=+338.226086821" Jul 29 14:06:48.237779 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:48.237732 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:06:48.238089 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:48.237829 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:06:49.032982 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:49.032944 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" event={"ID":"e33461ba-7347-4d89-91dc-e4d94c280cd5","Type":"ContainerStarted","Data":"650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818"} Jul 29 14:06:49.033494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:49.033023 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:49.034895 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:49.034863 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" event={"ID":"77b01920-4ef0-42f1-b160-c11a12fc72c8","Type":"ContainerStarted","Data":"efcd58e0e8c85739a5d8dad7b03aeb7e5db7cde99655a5c0fa05f8db34e25f31"} Jul 29 14:06:49.035130 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:49.035110 2560 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-5rjck container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 14:06:49.035197 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:49.035149 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" podUID="e33461ba-7347-4d89-91dc-e4d94c280cd5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 14:06:49.035371 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:49.035355 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:06:49.058735 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:49.058685 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" podStartSLOduration=1.454830984 podStartE2EDuration="6.058663155s" totalImagesPullingTime="4.603832171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:06:49.056395711 +0000 UTC m=+342.257896885" watchObservedRunningTime="2026-07-29 14:06:49.058663155 +0000 UTC m=+342.260164304" Jul 29 14:06:49.085157 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:49.085103 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" podStartSLOduration=2.053238887 podStartE2EDuration="7.085085901s" totalImagesPullingTime="5.031847014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:06:49.08307386 +0000 UTC m=+342.284575042" watchObservedRunningTime="2026-07-29 14:06:49.085085901 +0000 UTC m=+342.286587073" Jul 29 14:06:50.041493 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:50.041467 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:53.649701 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.649662 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm"] Jul 29 14:06:53.652888 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.652868 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:06:53.655663 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.655641 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 29 14:06:53.656012 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.655998 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8lzks\"" Jul 29 14:06:53.669802 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.669775 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs"] Jul 29 14:06:53.673133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.673114 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:06:53.676522 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.676494 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 29 14:06:53.676855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.676840 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ls872\"" Jul 29 14:06:53.685843 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.685820 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm"] Jul 29 14:06:53.706808 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.706779 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs"] Jul 29 14:06:53.810157 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.810128 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgwwh\" (UniqueName: \"kubernetes.io/projected/dff5f71f-1475-46e0-a2eb-e7676472a8ea-kube-api-access-kgwwh\") pod \"llmisvc-controller-manager-777c6bd7d5-zrfzs\" (UID: \"dff5f71f-1475-46e0-a2eb-e7676472a8ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:06:53.810357 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.810187 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jw8t\" (UniqueName: \"kubernetes.io/projected/ba573e67-60f3-4305-8459-e91d880b01a7-kube-api-access-2jw8t\") pod \"kserve-controller-manager-d6f4b65b8-ldbdm\" (UID: \"ba573e67-60f3-4305-8459-e91d880b01a7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:06:53.810357 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.810240 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dff5f71f-1475-46e0-a2eb-e7676472a8ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zrfzs\" (UID: \"dff5f71f-1475-46e0-a2eb-e7676472a8ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:06:53.810357 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.810348 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba573e67-60f3-4305-8459-e91d880b01a7-cert\") pod \"kserve-controller-manager-d6f4b65b8-ldbdm\" (UID: \"ba573e67-60f3-4305-8459-e91d880b01a7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:06:53.911806 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.911694 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgwwh\" (UniqueName: \"kubernetes.io/projected/dff5f71f-1475-46e0-a2eb-e7676472a8ea-kube-api-access-kgwwh\") pod \"llmisvc-controller-manager-777c6bd7d5-zrfzs\" (UID: \"dff5f71f-1475-46e0-a2eb-e7676472a8ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:06:53.911806 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.911802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jw8t\" (UniqueName: \"kubernetes.io/projected/ba573e67-60f3-4305-8459-e91d880b01a7-kube-api-access-2jw8t\") pod \"kserve-controller-manager-d6f4b65b8-ldbdm\" (UID: \"ba573e67-60f3-4305-8459-e91d880b01a7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:06:53.912026 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.911842 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dff5f71f-1475-46e0-a2eb-e7676472a8ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zrfzs\" (UID: \"dff5f71f-1475-46e0-a2eb-e7676472a8ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:06:53.912026 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.911908 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba573e67-60f3-4305-8459-e91d880b01a7-cert\") pod \"kserve-controller-manager-d6f4b65b8-ldbdm\" (UID: \"ba573e67-60f3-4305-8459-e91d880b01a7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:06:53.912026 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:06:53.911977 2560 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 29 14:06:53.912192 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:06:53.912037 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dff5f71f-1475-46e0-a2eb-e7676472a8ea-cert podName:dff5f71f-1475-46e0-a2eb-e7676472a8ea nodeName:}" failed. No retries permitted until 2026-07-29 14:06:54.41202106 +0000 UTC m=+347.613522211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dff5f71f-1475-46e0-a2eb-e7676472a8ea-cert") pod "llmisvc-controller-manager-777c6bd7d5-zrfzs" (UID: "dff5f71f-1475-46e0-a2eb-e7676472a8ea") : secret "llmisvc-webhook-server-cert" not found Jul 29 14:06:53.915057 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.915033 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba573e67-60f3-4305-8459-e91d880b01a7-cert\") pod \"kserve-controller-manager-d6f4b65b8-ldbdm\" (UID: \"ba573e67-60f3-4305-8459-e91d880b01a7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:06:53.928306 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.928267 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgwwh\" (UniqueName: \"kubernetes.io/projected/dff5f71f-1475-46e0-a2eb-e7676472a8ea-kube-api-access-kgwwh\") pod \"llmisvc-controller-manager-777c6bd7d5-zrfzs\" (UID: \"dff5f71f-1475-46e0-a2eb-e7676472a8ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:06:53.928443 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.928271 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jw8t\" (UniqueName: \"kubernetes.io/projected/ba573e67-60f3-4305-8459-e91d880b01a7-kube-api-access-2jw8t\") pod \"kserve-controller-manager-d6f4b65b8-ldbdm\" (UID: \"ba573e67-60f3-4305-8459-e91d880b01a7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:06:53.964155 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:53.964118 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:06:54.113288 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:54.113256 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm"] Jul 29 14:06:54.117118 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:06:54.117094 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba573e67_60f3_4305_8459_e91d880b01a7.slice/crio-4856958dad0272e4bb913002b704b32ec9c96e40c4b29e7bccf04a6283ec5c10 WatchSource:0}: Error finding container 4856958dad0272e4bb913002b704b32ec9c96e40c4b29e7bccf04a6283ec5c10: Status 404 returned error can't find the container with id 4856958dad0272e4bb913002b704b32ec9c96e40c4b29e7bccf04a6283ec5c10 Jul 29 14:06:54.416640 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:54.416598 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dff5f71f-1475-46e0-a2eb-e7676472a8ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zrfzs\" (UID: \"dff5f71f-1475-46e0-a2eb-e7676472a8ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:06:54.419771 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:54.419750 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dff5f71f-1475-46e0-a2eb-e7676472a8ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zrfzs\" (UID: \"dff5f71f-1475-46e0-a2eb-e7676472a8ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:06:54.583894 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:54.583851 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:06:54.730325 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:54.730301 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs"] Jul 29 14:06:54.732082 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:06:54.732056 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddff5f71f_1475_46e0_a2eb_e7676472a8ea.slice/crio-fdcb6dd82b0c4259adef635620915310247bab42188fce689669031995e70f9b WatchSource:0}: Error finding container fdcb6dd82b0c4259adef635620915310247bab42188fce689669031995e70f9b: Status 404 returned error can't find the container with id fdcb6dd82b0c4259adef635620915310247bab42188fce689669031995e70f9b Jul 29 14:06:55.067623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:55.067575 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" event={"ID":"ba573e67-60f3-4305-8459-e91d880b01a7","Type":"ContainerStarted","Data":"4856958dad0272e4bb913002b704b32ec9c96e40c4b29e7bccf04a6283ec5c10"} Jul 29 14:06:55.068958 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:55.068931 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" event={"ID":"dff5f71f-1475-46e0-a2eb-e7676472a8ea","Type":"ContainerStarted","Data":"fdcb6dd82b0c4259adef635620915310247bab42188fce689669031995e70f9b"} Jul 29 14:06:57.664272 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:57.664234 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck"] Jul 29 14:06:57.664793 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:57.664467 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" podUID="e33461ba-7347-4d89-91dc-e4d94c280cd5" containerName="discovery" containerID="cri-o://650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818" gracePeriod=30 Jul 29 14:06:57.920555 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:57.920520 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:58.050746 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.050710 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4f9c\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-kube-api-access-p4f9c\") pod \"e33461ba-7347-4d89-91dc-e4d94c280cd5\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " Jul 29 14:06:58.050911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.050761 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-token\") pod \"e33461ba-7347-4d89-91dc-e4d94c280cd5\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " Jul 29 14:06:58.050911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.050807 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-dns-cert\") pod \"e33461ba-7347-4d89-91dc-e4d94c280cd5\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " Jul 29 14:06:58.050911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.050858 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33461ba-7347-4d89-91dc-e4d94c280cd5-local-certs\") pod \"e33461ba-7347-4d89-91dc-e4d94c280cd5\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " Jul 29 14:06:58.050911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.050884 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-ca-configmap\") pod \"e33461ba-7347-4d89-91dc-e4d94c280cd5\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " Jul 29 14:06:58.051157 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.050958 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-kubeconfig\") pod \"e33461ba-7347-4d89-91dc-e4d94c280cd5\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " Jul 29 14:06:58.051157 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.051004 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-cacerts\") pod \"e33461ba-7347-4d89-91dc-e4d94c280cd5\" (UID: \"e33461ba-7347-4d89-91dc-e4d94c280cd5\") " Jul 29 14:06:58.051607 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.051527 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e33461ba-7347-4d89-91dc-e4d94c280cd5" (UID: "e33461ba-7347-4d89-91dc-e4d94c280cd5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:06:58.054877 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.053947 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e33461ba-7347-4d89-91dc-e4d94c280cd5" (UID: "e33461ba-7347-4d89-91dc-e4d94c280cd5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:06:58.054877 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.054753 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33461ba-7347-4d89-91dc-e4d94c280cd5-local-certs" (OuterVolumeSpecName: "local-certs") pod "e33461ba-7347-4d89-91dc-e4d94c280cd5" (UID: "e33461ba-7347-4d89-91dc-e4d94c280cd5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:06:58.054877 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.054868 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-kube-api-access-p4f9c" (OuterVolumeSpecName: "kube-api-access-p4f9c") pod "e33461ba-7347-4d89-91dc-e4d94c280cd5" (UID: "e33461ba-7347-4d89-91dc-e4d94c280cd5"). InnerVolumeSpecName "kube-api-access-p4f9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:06:58.055099 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.054888 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-cacerts" (OuterVolumeSpecName: "cacerts") pod "e33461ba-7347-4d89-91dc-e4d94c280cd5" (UID: "e33461ba-7347-4d89-91dc-e4d94c280cd5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:06:58.055296 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.055266 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e33461ba-7347-4d89-91dc-e4d94c280cd5" (UID: "e33461ba-7347-4d89-91dc-e4d94c280cd5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:06:58.055583 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.055497 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "e33461ba-7347-4d89-91dc-e4d94c280cd5" (UID: "e33461ba-7347-4d89-91dc-e4d94c280cd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:06:58.088742 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.088483 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" event={"ID":"ba573e67-60f3-4305-8459-e91d880b01a7","Type":"ContainerStarted","Data":"cc4618d958c1d37577f83ae046ef713cf137df14832faf4237e3f7f2dae0c913"} Jul 29 14:06:58.088742 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.088580 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:06:58.090668 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.090640 2560 generic.go:356] "Generic (PLEG): container finished" podID="e33461ba-7347-4d89-91dc-e4d94c280cd5" containerID="650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818" exitCode=0 Jul 29 14:06:58.090668 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.090670 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" event={"ID":"e33461ba-7347-4d89-91dc-e4d94c280cd5","Type":"ContainerDied","Data":"650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818"} Jul 29 14:06:58.090863 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.090692 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" Jul 29 14:06:58.090863 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.090730 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck" event={"ID":"e33461ba-7347-4d89-91dc-e4d94c280cd5","Type":"ContainerDied","Data":"6a55129b2799ba9333e2076b1237a467d43642d16abe537e724c97127ba2d57d"} Jul 29 14:06:58.090863 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.090756 2560 scope.go:117] "RemoveContainer" containerID="650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818" Jul 29 14:06:58.102180 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.102163 2560 scope.go:117] "RemoveContainer" containerID="650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818" Jul 29 14:06:58.102439 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:06:58.102424 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818\": container with ID starting with 650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818 not found: ID does not exist" containerID="650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818" Jul 29 14:06:58.102501 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.102445 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818"} err="failed to get container status \"650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818\": rpc error: code = NotFound desc = could not find container \"650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818\": container with ID starting with 650dd2e975f81f4cd22dda75a453a0fb8046a46278c3f13603514f7aafd55818 not found: ID does not exist" Jul 29 14:06:58.139556 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.139489 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" podStartSLOduration=2.397048504 podStartE2EDuration="5.139472217s" totalImagesPullingTime="2.742423713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:06:58.133098044 +0000 UTC m=+351.334599217" watchObservedRunningTime="2026-07-29 14:06:58.139472217 +0000 UTC m=+351.340973390" Jul 29 14:06:58.152190 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.152157 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck"] Jul 29 14:06:58.152472 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.152454 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4f9c\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-kube-api-access-p4f9c\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:58.152519 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.152477 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-token\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:58.152519 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.152487 2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-dns-cert\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:58.152519 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.152495 2560 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33461ba-7347-4d89-91dc-e4d94c280cd5-local-certs\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:58.152519 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.152504 2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-csr-ca-configmap\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:58.152519 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.152512 2560 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-istio-kubeconfig\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:58.152684 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.152521 2560 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e33461ba-7347-4d89-91dc-e4d94c280cd5-cacerts\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:06:58.156894 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:58.156870 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5rjck"] Jul 29 14:06:59.384765 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:06:59.384718 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33461ba-7347-4d89-91dc-e4d94c280cd5" path="/var/lib/kubelet/pods/e33461ba-7347-4d89-91dc-e4d94c280cd5/volumes" Jul 29 14:07:00.106641 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:00.106596 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" event={"ID":"dff5f71f-1475-46e0-a2eb-e7676472a8ea","Type":"ContainerStarted","Data":"4f6a5e69d18f3a310d75654b2daa93cd4a3d08c994849c22ebfa926158b45348"} Jul 29 14:07:00.106836 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:00.106763 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:07:00.126888 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:00.126823 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" podStartSLOduration=2.555178942 podStartE2EDuration="7.126803816s" totalImagesPullingTime="4.571624874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:07:00.126297553 +0000 UTC m=+353.327798725" watchObservedRunningTime="2026-07-29 14:07:00.126803816 +0000 UTC m=+353.328304991" Jul 29 14:07:01.048398 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:01.048369 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mhpqh" Jul 29 14:07:29.104470 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:29.104439 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ldbdm" Jul 29 14:07:31.114987 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:31.114948 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zrfzs" Jul 29 14:07:42.627337 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.627285 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9fpdc"] Jul 29 14:07:42.627777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.627677 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33461ba-7347-4d89-91dc-e4d94c280cd5" containerName="discovery" Jul 29 14:07:42.627777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.627688 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33461ba-7347-4d89-91dc-e4d94c280cd5" containerName="discovery" Jul 29 14:07:42.627777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.627762 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33461ba-7347-4d89-91dc-e4d94c280cd5" containerName="discovery" Jul 29 14:07:42.631951 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.631928 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" Jul 29 14:07:42.634421 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.634405 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 14:07:42.634517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.634475 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 14:07:42.634517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.634408 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-2sj7t\"" Jul 29 14:07:42.641007 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.640824 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9fpdc"] Jul 29 14:07:42.701982 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.701950 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqk9l\" (UniqueName: \"kubernetes.io/projected/0fe29968-4f5a-493e-b76a-20ba4da86fc8-kube-api-access-dqk9l\") pod \"kuadrant-operator-catalog-9fpdc\" (UID: \"0fe29968-4f5a-493e-b76a-20ba4da86fc8\") " pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" Jul 29 14:07:42.802829 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.802792 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqk9l\" (UniqueName: \"kubernetes.io/projected/0fe29968-4f5a-493e-b76a-20ba4da86fc8-kube-api-access-dqk9l\") pod \"kuadrant-operator-catalog-9fpdc\" (UID: \"0fe29968-4f5a-493e-b76a-20ba4da86fc8\") " pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" Jul 29 14:07:42.811395 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.811368 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqk9l\" (UniqueName: \"kubernetes.io/projected/0fe29968-4f5a-493e-b76a-20ba4da86fc8-kube-api-access-dqk9l\") pod \"kuadrant-operator-catalog-9fpdc\" (UID: \"0fe29968-4f5a-493e-b76a-20ba4da86fc8\") " pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" Jul 29 14:07:42.943605 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.943577 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" Jul 29 14:07:42.970020 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:42.969988 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9fpdc"] Jul 29 14:07:43.098018 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.097987 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9fpdc"] Jul 29 14:07:43.101305 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:07:43.101267 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe29968_4f5a_493e_b76a_20ba4da86fc8.slice/crio-f0e0b8dd536ed54a38ed431c90508e80a8528e8658d55c07af53373119ecf129 WatchSource:0}: Error finding container f0e0b8dd536ed54a38ed431c90508e80a8528e8658d55c07af53373119ecf129: Status 404 returned error can't find the container with id f0e0b8dd536ed54a38ed431c90508e80a8528e8658d55c07af53373119ecf129 Jul 29 14:07:43.184232 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.184202 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rn5jh"] Jul 29 14:07:43.186936 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.186914 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" Jul 29 14:07:43.195872 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.195845 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rn5jh"] Jul 29 14:07:43.307128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.307090 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2bfh\" (UniqueName: \"kubernetes.io/projected/b810810c-028e-473e-b761-7e1db11b727a-kube-api-access-h2bfh\") pod \"kuadrant-operator-catalog-rn5jh\" (UID: \"b810810c-028e-473e-b761-7e1db11b727a\") " pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" Jul 29 14:07:43.315373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.315345 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" event={"ID":"0fe29968-4f5a-493e-b76a-20ba4da86fc8","Type":"ContainerStarted","Data":"f0e0b8dd536ed54a38ed431c90508e80a8528e8658d55c07af53373119ecf129"} Jul 29 14:07:43.407843 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.407803 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2bfh\" (UniqueName: \"kubernetes.io/projected/b810810c-028e-473e-b761-7e1db11b727a-kube-api-access-h2bfh\") pod \"kuadrant-operator-catalog-rn5jh\" (UID: \"b810810c-028e-473e-b761-7e1db11b727a\") " pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" Jul 29 14:07:43.420428 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.420402 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2bfh\" (UniqueName: \"kubernetes.io/projected/b810810c-028e-473e-b761-7e1db11b727a-kube-api-access-h2bfh\") pod \"kuadrant-operator-catalog-rn5jh\" (UID: \"b810810c-028e-473e-b761-7e1db11b727a\") " pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" Jul 29 14:07:43.498219 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.498137 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" Jul 29 14:07:43.639502 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:43.639455 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rn5jh"] Jul 29 14:07:43.641686 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:07:43.641657 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb810810c_028e_473e_b761_7e1db11b727a.slice/crio-98b2cd8b87e5ac7badfd32a787ba8d201e6b6a6cf423fe74e98ab4aeb9e574dd WatchSource:0}: Error finding container 98b2cd8b87e5ac7badfd32a787ba8d201e6b6a6cf423fe74e98ab4aeb9e574dd: Status 404 returned error can't find the container with id 98b2cd8b87e5ac7badfd32a787ba8d201e6b6a6cf423fe74e98ab4aeb9e574dd Jul 29 14:07:44.322137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:44.322105 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" event={"ID":"b810810c-028e-473e-b761-7e1db11b727a","Type":"ContainerStarted","Data":"98b2cd8b87e5ac7badfd32a787ba8d201e6b6a6cf423fe74e98ab4aeb9e574dd"} Jul 29 14:07:45.329018 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:45.328911 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" event={"ID":"0fe29968-4f5a-493e-b76a-20ba4da86fc8","Type":"ContainerStarted","Data":"69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21"} Jul 29 14:07:45.329018 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:45.328961 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" podUID="0fe29968-4f5a-493e-b76a-20ba4da86fc8" containerName="registry-server" containerID="cri-o://69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21" gracePeriod=2 Jul 29 14:07:45.330490 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:45.330467 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" event={"ID":"b810810c-028e-473e-b761-7e1db11b727a","Type":"ContainerStarted","Data":"765317a91cf54dad14b36aad029adf881db59b3ee47298aff60b8b159cdb62f8"} Jul 29 14:07:45.343464 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:45.343406 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" podStartSLOduration=1.438431869 podStartE2EDuration="3.343388416s" totalImagesPullingTime="1.904956547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:07:45.343286102 +0000 UTC m=+398.544787278" watchObservedRunningTime="2026-07-29 14:07:45.343388416 +0000 UTC m=+398.544889590" Jul 29 14:07:45.358660 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:45.358621 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" podStartSLOduration=0.997497349 podStartE2EDuration="2.35860695s" totalImagesPullingTime="1.361109601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:07:45.357067828 +0000 UTC m=+398.558569229" watchObservedRunningTime="2026-07-29 14:07:45.35860695 +0000 UTC m=+398.560108121" Jul 29 14:07:45.576524 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:45.576500 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" Jul 29 14:07:45.628494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:45.628414 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqk9l\" (UniqueName: \"kubernetes.io/projected/0fe29968-4f5a-493e-b76a-20ba4da86fc8-kube-api-access-dqk9l\") pod \"0fe29968-4f5a-493e-b76a-20ba4da86fc8\" (UID: \"0fe29968-4f5a-493e-b76a-20ba4da86fc8\") " Jul 29 14:07:45.630741 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:45.630718 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe29968-4f5a-493e-b76a-20ba4da86fc8-kube-api-access-dqk9l" (OuterVolumeSpecName: "kube-api-access-dqk9l") pod "0fe29968-4f5a-493e-b76a-20ba4da86fc8" (UID: "0fe29968-4f5a-493e-b76a-20ba4da86fc8"). InnerVolumeSpecName "kube-api-access-dqk9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:07:45.729562 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:45.729510 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqk9l\" (UniqueName: \"kubernetes.io/projected/0fe29968-4f5a-493e-b76a-20ba4da86fc8-kube-api-access-dqk9l\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:07:46.336875 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:46.336785 2560 generic.go:356] "Generic (PLEG): container finished" podID="0fe29968-4f5a-493e-b76a-20ba4da86fc8" containerID="69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21" exitCode=0 Jul 29 14:07:46.336875 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:46.336829 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" Jul 29 14:07:46.337388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:46.336874 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" event={"ID":"0fe29968-4f5a-493e-b76a-20ba4da86fc8","Type":"ContainerDied","Data":"69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21"} Jul 29 14:07:46.337388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:46.336926 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9fpdc" event={"ID":"0fe29968-4f5a-493e-b76a-20ba4da86fc8","Type":"ContainerDied","Data":"f0e0b8dd536ed54a38ed431c90508e80a8528e8658d55c07af53373119ecf129"} Jul 29 14:07:46.337388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:46.336947 2560 scope.go:117] "RemoveContainer" containerID="69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21" Jul 29 14:07:46.346560 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:46.346524 2560 scope.go:117] "RemoveContainer" containerID="69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21" Jul 29 14:07:46.346806 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:07:46.346789 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21\": container with ID starting with 69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21 not found: ID does not exist" containerID="69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21" Jul 29 14:07:46.346851 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:46.346815 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21"} err="failed to get container status \"69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21\": rpc error: code = NotFound desc = could not find container \"69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21\": container with ID starting with 69820dbce8934b842c3dea7d27a6f529cf394c13811b988250aa657e11c83c21 not found: ID does not exist" Jul 29 14:07:46.357899 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:46.357873 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9fpdc"] Jul 29 14:07:46.361604 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:46.361586 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9fpdc"] Jul 29 14:07:47.380194 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:47.380159 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fe29968-4f5a-493e-b76a-20ba4da86fc8" path="/var/lib/kubelet/pods/0fe29968-4f5a-493e-b76a-20ba4da86fc8/volumes" Jul 29 14:07:53.499224 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:53.499197 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" Jul 29 14:07:53.499654 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:53.499247 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" Jul 29 14:07:53.520924 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:53.520900 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" Jul 29 14:07:54.398087 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:54.398057 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rn5jh" Jul 29 14:07:58.195059 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.195024 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm"] Jul 29 14:07:58.195437 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.195362 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe29968-4f5a-493e-b76a-20ba4da86fc8" containerName="registry-server" Jul 29 14:07:58.195437 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.195373 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe29968-4f5a-493e-b76a-20ba4da86fc8" containerName="registry-server" Jul 29 14:07:58.195512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.195439 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe29968-4f5a-493e-b76a-20ba4da86fc8" containerName="registry-server" Jul 29 14:07:58.198512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.198495 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.200888 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.200871 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fcttf\"" Jul 29 14:07:58.206880 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.206858 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm"] Jul 29 14:07:58.335311 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.335268 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.335500 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.335321 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.335500 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.335421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8tw\" (UniqueName: \"kubernetes.io/projected/96d924bb-eacb-4e53-a8f8-89fe595a9d69-kube-api-access-dj8tw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.436563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.436497 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8tw\" (UniqueName: \"kubernetes.io/projected/96d924bb-eacb-4e53-a8f8-89fe595a9d69-kube-api-access-dj8tw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.436765 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.436598 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.436765 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.436627 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.436995 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.436980 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.437042 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.437012 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.445162 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.445110 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8tw\" (UniqueName: \"kubernetes.io/projected/96d924bb-eacb-4e53-a8f8-89fe595a9d69-kube-api-access-dj8tw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.509120 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.509085 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:07:58.642945 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.642923 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm"] Jul 29 14:07:58.644502 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:07:58.644460 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d924bb_eacb_4e53_a8f8_89fe595a9d69.slice/crio-1b3ddf8d5018eabf19c8faa66346bd2d870ed362491f21929d14c908d808be82 WatchSource:0}: Error finding container 1b3ddf8d5018eabf19c8faa66346bd2d870ed362491f21929d14c908d808be82: Status 404 returned error can't find the container with id 1b3ddf8d5018eabf19c8faa66346bd2d870ed362491f21929d14c908d808be82 Jul 29 14:07:58.991486 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.991401 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx"] Jul 29 14:07:58.995027 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:58.995008 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.002391 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.002280 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx"] Jul 29 14:07:59.143485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.143454 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cwd2\" (UniqueName: \"kubernetes.io/projected/839a6e05-d66f-4844-a6b6-144db923302b-kube-api-access-7cwd2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.143485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.143487 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.143801 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.143512 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.244373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.244284 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cwd2\" (UniqueName: \"kubernetes.io/projected/839a6e05-d66f-4844-a6b6-144db923302b-kube-api-access-7cwd2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.244373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.244323 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.244373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.244351 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.244841 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.244669 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.244841 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.244744 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.253203 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.253183 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cwd2\" (UniqueName: \"kubernetes.io/projected/839a6e05-d66f-4844-a6b6-144db923302b-kube-api-access-7cwd2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.305144 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.305115 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:07:59.403344 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.403310 2560 generic.go:356] "Generic (PLEG): container finished" podID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerID="b19fc78c4a0bf22fd79a8090ce778098798123f3e2f315adc75f9c92114f8159" exitCode=0 Jul 29 14:07:59.403506 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.403371 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" event={"ID":"96d924bb-eacb-4e53-a8f8-89fe595a9d69","Type":"ContainerDied","Data":"b19fc78c4a0bf22fd79a8090ce778098798123f3e2f315adc75f9c92114f8159"} Jul 29 14:07:59.403506 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.403416 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" event={"ID":"96d924bb-eacb-4e53-a8f8-89fe595a9d69","Type":"ContainerStarted","Data":"1b3ddf8d5018eabf19c8faa66346bd2d870ed362491f21929d14c908d808be82"} Jul 29 14:07:59.444019 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.443995 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx"] Jul 29 14:07:59.446491 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:07:59.446465 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839a6e05_d66f_4844_a6b6_144db923302b.slice/crio-a6805946e87fff0b2a8fddb2974d5fb61b4c9fd97e1f176254bbb5e6d34e4a70 WatchSource:0}: Error finding container a6805946e87fff0b2a8fddb2974d5fb61b4c9fd97e1f176254bbb5e6d34e4a70: Status 404 returned error can't find the container with id a6805946e87fff0b2a8fddb2974d5fb61b4c9fd97e1f176254bbb5e6d34e4a70 Jul 29 14:07:59.789046 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.788960 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556"] Jul 29 14:07:59.792657 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.792639 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:07:59.800359 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.800333 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556"] Jul 29 14:07:59.850620 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.850584 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:07:59.850806 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.850659 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcp46\" (UniqueName: \"kubernetes.io/projected/3301ee40-54e0-47b9-94f1-357d7763fb4f-kube-api-access-gcp46\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:07:59.850806 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.850768 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:07:59.951759 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.951716 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:07:59.951944 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.951787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:07:59.951944 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.951841 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcp46\" (UniqueName: \"kubernetes.io/projected/3301ee40-54e0-47b9-94f1-357d7763fb4f-kube-api-access-gcp46\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:07:59.952096 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.952077 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:07:59.952153 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.952120 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:07:59.960365 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:07:59.960336 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcp46\" (UniqueName: \"kubernetes.io/projected/3301ee40-54e0-47b9-94f1-357d7763fb4f-kube-api-access-gcp46\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:08:00.102728 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.102642 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:08:00.234971 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.234946 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556"] Jul 29 14:08:00.236583 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:00.236559 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3301ee40_54e0_47b9_94f1_357d7763fb4f.slice/crio-3d9283c5257100d9d4dabc0c2c32dfe7ae404734041d588efe1aa82912140234 WatchSource:0}: Error finding container 3d9283c5257100d9d4dabc0c2c32dfe7ae404734041d588efe1aa82912140234: Status 404 returned error can't find the container with id 3d9283c5257100d9d4dabc0c2c32dfe7ae404734041d588efe1aa82912140234 Jul 29 14:08:00.389865 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.389786 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh"] Jul 29 14:08:00.393394 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.393376 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.400489 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.400465 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh"] Jul 29 14:08:00.411230 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.411208 2560 generic.go:356] "Generic (PLEG): container finished" podID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerID="6ae2a8ad4c5c7ed2b0e16dba110def359e514cb8489ca3c53a26aca15c6500d7" exitCode=0 Jul 29 14:08:00.411364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.411253 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" event={"ID":"3301ee40-54e0-47b9-94f1-357d7763fb4f","Type":"ContainerDied","Data":"6ae2a8ad4c5c7ed2b0e16dba110def359e514cb8489ca3c53a26aca15c6500d7"} Jul 29 14:08:00.411364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.411310 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" event={"ID":"3301ee40-54e0-47b9-94f1-357d7763fb4f","Type":"ContainerStarted","Data":"3d9283c5257100d9d4dabc0c2c32dfe7ae404734041d588efe1aa82912140234"} Jul 29 14:08:00.413238 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.413218 2560 generic.go:356] "Generic (PLEG): container finished" podID="839a6e05-d66f-4844-a6b6-144db923302b" containerID="9a0b08140f2451cc51b2d34a512503b3bd559d43341d55fe66ec2552190df445" exitCode=0 Jul 29 14:08:00.413348 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.413262 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" event={"ID":"839a6e05-d66f-4844-a6b6-144db923302b","Type":"ContainerDied","Data":"9a0b08140f2451cc51b2d34a512503b3bd559d43341d55fe66ec2552190df445"} Jul 29 14:08:00.413348 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.413296 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" event={"ID":"839a6e05-d66f-4844-a6b6-144db923302b","Type":"ContainerStarted","Data":"a6805946e87fff0b2a8fddb2974d5fb61b4c9fd97e1f176254bbb5e6d34e4a70"} Jul 29 14:08:00.456440 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.456414 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.456564 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.456500 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.456564 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.456530 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7gp5\" (UniqueName: \"kubernetes.io/projected/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-kube-api-access-w7gp5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.557880 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.557826 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.558045 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.557900 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7gp5\" (UniqueName: \"kubernetes.io/projected/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-kube-api-access-w7gp5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.558045 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.557972 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.558252 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.558233 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.558330 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.558313 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.566384 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.566364 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7gp5\" (UniqueName: \"kubernetes.io/projected/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-kube-api-access-w7gp5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.712893 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.712866 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:00.847571 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:00.847527 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh"] Jul 29 14:08:00.850252 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:00.850224 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0151f9_7a05_44e3_b38a_3d6d96b7363e.slice/crio-0fde20e01e5bf518046e8bbefb71e147e580f674c5f356c2fe4a2cb48627676e WatchSource:0}: Error finding container 0fde20e01e5bf518046e8bbefb71e147e580f674c5f356c2fe4a2cb48627676e: Status 404 returned error can't find the container with id 0fde20e01e5bf518046e8bbefb71e147e580f674c5f356c2fe4a2cb48627676e Jul 29 14:08:01.424608 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:01.424571 2560 generic.go:356] "Generic (PLEG): container finished" podID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerID="462635a4b61e845093f140d7f14bc66ce57c2f623202cc900b8e6e0c6d9d7dc8" exitCode=0 Jul 29 14:08:01.425042 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:01.424708 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" event={"ID":"fb0151f9-7a05-44e3-b38a-3d6d96b7363e","Type":"ContainerDied","Data":"462635a4b61e845093f140d7f14bc66ce57c2f623202cc900b8e6e0c6d9d7dc8"} Jul 29 14:08:01.425042 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:01.424793 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" event={"ID":"fb0151f9-7a05-44e3-b38a-3d6d96b7363e","Type":"ContainerStarted","Data":"0fde20e01e5bf518046e8bbefb71e147e580f674c5f356c2fe4a2cb48627676e"} Jul 29 14:08:01.426656 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:01.426632 2560 generic.go:356] "Generic (PLEG): container finished" podID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerID="d206993626b6e06523f0a25556bca9e2e9c4f0a43fa86ab99f5014992fb9fd5a" exitCode=0 Jul 29 14:08:01.426776 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:01.426715 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" event={"ID":"3301ee40-54e0-47b9-94f1-357d7763fb4f","Type":"ContainerDied","Data":"d206993626b6e06523f0a25556bca9e2e9c4f0a43fa86ab99f5014992fb9fd5a"} Jul 29 14:08:02.433181 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:02.433149 2560 generic.go:356] "Generic (PLEG): container finished" podID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerID="dce153aed503752e2ae7c750e046efc4d35260714f022d74c4316ee470deedae" exitCode=0 Jul 29 14:08:02.433652 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:02.433236 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" event={"ID":"fb0151f9-7a05-44e3-b38a-3d6d96b7363e","Type":"ContainerDied","Data":"dce153aed503752e2ae7c750e046efc4d35260714f022d74c4316ee470deedae"} Jul 29 14:08:02.435508 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:02.435489 2560 generic.go:356] "Generic (PLEG): container finished" podID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerID="761d407a8c1bfcf7010728d2d0be7aa4d7079e494a294622a5514dc3c63ae68e" exitCode=0 Jul 29 14:08:02.435628 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:02.435580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" event={"ID":"3301ee40-54e0-47b9-94f1-357d7763fb4f","Type":"ContainerDied","Data":"761d407a8c1bfcf7010728d2d0be7aa4d7079e494a294622a5514dc3c63ae68e"} Jul 29 14:08:02.437408 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:02.437387 2560 generic.go:356] "Generic (PLEG): container finished" podID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerID="ace7327e2e146b86341b1137680b2cc9ccb5aa881c5c962b559a4a22bb8785c4" exitCode=0 Jul 29 14:08:02.437511 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:02.437432 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" event={"ID":"96d924bb-eacb-4e53-a8f8-89fe595a9d69","Type":"ContainerDied","Data":"ace7327e2e146b86341b1137680b2cc9ccb5aa881c5c962b559a4a22bb8785c4"} Jul 29 14:08:03.444284 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.444254 2560 generic.go:356] "Generic (PLEG): container finished" podID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerID="03f68d8aef0edf1651bb7c2841288b227c31620871f2836b1684de8efe36bc33" exitCode=0 Jul 29 14:08:03.444736 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.444340 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" event={"ID":"96d924bb-eacb-4e53-a8f8-89fe595a9d69","Type":"ContainerDied","Data":"03f68d8aef0edf1651bb7c2841288b227c31620871f2836b1684de8efe36bc33"} Jul 29 14:08:03.446551 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.446513 2560 generic.go:356] "Generic (PLEG): container finished" podID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerID="255e988c37bf77ac12dec280c57120357be51de124032e1c0901c97e3c39b9a4" exitCode=0 Jul 29 14:08:03.446670 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.446566 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" event={"ID":"fb0151f9-7a05-44e3-b38a-3d6d96b7363e","Type":"ContainerDied","Data":"255e988c37bf77ac12dec280c57120357be51de124032e1c0901c97e3c39b9a4"} Jul 29 14:08:03.574580 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.574559 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:08:03.688526 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.688491 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcp46\" (UniqueName: \"kubernetes.io/projected/3301ee40-54e0-47b9-94f1-357d7763fb4f-kube-api-access-gcp46\") pod \"3301ee40-54e0-47b9-94f1-357d7763fb4f\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " Jul 29 14:08:03.688717 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.688533 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-bundle\") pod \"3301ee40-54e0-47b9-94f1-357d7763fb4f\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " Jul 29 14:08:03.688717 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.688671 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-util\") pod \"3301ee40-54e0-47b9-94f1-357d7763fb4f\" (UID: \"3301ee40-54e0-47b9-94f1-357d7763fb4f\") " Jul 29 14:08:03.689016 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.688991 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-bundle" (OuterVolumeSpecName: "bundle") pod "3301ee40-54e0-47b9-94f1-357d7763fb4f" (UID: "3301ee40-54e0-47b9-94f1-357d7763fb4f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:08:03.690846 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.690824 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3301ee40-54e0-47b9-94f1-357d7763fb4f-kube-api-access-gcp46" (OuterVolumeSpecName: "kube-api-access-gcp46") pod "3301ee40-54e0-47b9-94f1-357d7763fb4f" (UID: "3301ee40-54e0-47b9-94f1-357d7763fb4f"). InnerVolumeSpecName "kube-api-access-gcp46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:08:03.694119 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.694083 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-util" (OuterVolumeSpecName: "util") pod "3301ee40-54e0-47b9-94f1-357d7763fb4f" (UID: "3301ee40-54e0-47b9-94f1-357d7763fb4f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:08:03.790184 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.790154 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcp46\" (UniqueName: \"kubernetes.io/projected/3301ee40-54e0-47b9-94f1-357d7763fb4f-kube-api-access-gcp46\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:03.790184 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.790177 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-bundle\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:03.790184 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:03.790187 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3301ee40-54e0-47b9-94f1-357d7763fb4f-util\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:04.453364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.453340 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" Jul 29 14:08:04.453364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.453354 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556" event={"ID":"3301ee40-54e0-47b9-94f1-357d7763fb4f","Type":"ContainerDied","Data":"3d9283c5257100d9d4dabc0c2c32dfe7ae404734041d588efe1aa82912140234"} Jul 29 14:08:04.453859 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.453406 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d9283c5257100d9d4dabc0c2c32dfe7ae404734041d588efe1aa82912140234" Jul 29 14:08:04.455279 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.455257 2560 generic.go:356] "Generic (PLEG): container finished" podID="839a6e05-d66f-4844-a6b6-144db923302b" containerID="438e2e64715f2401058af1b4fcfa042bdd19aa7094af6de53bab2a7ee10ebcc4" exitCode=0 Jul 29 14:08:04.455410 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.455324 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" event={"ID":"839a6e05-d66f-4844-a6b6-144db923302b","Type":"ContainerDied","Data":"438e2e64715f2401058af1b4fcfa042bdd19aa7094af6de53bab2a7ee10ebcc4"} Jul 29 14:08:04.632563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.632517 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:04.680350 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.680323 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:08:04.700196 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.700159 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-util\") pod \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " Jul 29 14:08:04.700905 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.700668 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7gp5\" (UniqueName: \"kubernetes.io/projected/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-kube-api-access-w7gp5\") pod \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " Jul 29 14:08:04.700905 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.700758 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-bundle\") pod \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\" (UID: \"fb0151f9-7a05-44e3-b38a-3d6d96b7363e\") " Jul 29 14:08:04.702144 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.702120 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-bundle" (OuterVolumeSpecName: "bundle") pod "fb0151f9-7a05-44e3-b38a-3d6d96b7363e" (UID: "fb0151f9-7a05-44e3-b38a-3d6d96b7363e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:08:04.703886 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.703824 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-kube-api-access-w7gp5" (OuterVolumeSpecName: "kube-api-access-w7gp5") pod "fb0151f9-7a05-44e3-b38a-3d6d96b7363e" (UID: "fb0151f9-7a05-44e3-b38a-3d6d96b7363e"). InnerVolumeSpecName "kube-api-access-w7gp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:08:04.708211 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.708184 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-util" (OuterVolumeSpecName: "util") pod "fb0151f9-7a05-44e3-b38a-3d6d96b7363e" (UID: "fb0151f9-7a05-44e3-b38a-3d6d96b7363e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:08:04.802249 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.802219 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-util\") pod \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " Jul 29 14:08:04.802397 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.802257 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-bundle\") pod \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " Jul 29 14:08:04.802397 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.802291 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj8tw\" (UniqueName: \"kubernetes.io/projected/96d924bb-eacb-4e53-a8f8-89fe595a9d69-kube-api-access-dj8tw\") pod \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\" (UID: \"96d924bb-eacb-4e53-a8f8-89fe595a9d69\") " Jul 29 14:08:04.802557 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.802517 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-bundle\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:04.802633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.802562 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-util\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:04.802633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.802578 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7gp5\" (UniqueName: \"kubernetes.io/projected/fb0151f9-7a05-44e3-b38a-3d6d96b7363e-kube-api-access-w7gp5\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:04.803003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.802975 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-bundle" (OuterVolumeSpecName: "bundle") pod "96d924bb-eacb-4e53-a8f8-89fe595a9d69" (UID: "96d924bb-eacb-4e53-a8f8-89fe595a9d69"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:08:04.804622 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.804593 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d924bb-eacb-4e53-a8f8-89fe595a9d69-kube-api-access-dj8tw" (OuterVolumeSpecName: "kube-api-access-dj8tw") pod "96d924bb-eacb-4e53-a8f8-89fe595a9d69" (UID: "96d924bb-eacb-4e53-a8f8-89fe595a9d69"). InnerVolumeSpecName "kube-api-access-dj8tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:08:04.807204 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.807172 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-util" (OuterVolumeSpecName: "util") pod "96d924bb-eacb-4e53-a8f8-89fe595a9d69" (UID: "96d924bb-eacb-4e53-a8f8-89fe595a9d69"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:08:04.903485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.903443 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-util\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:04.903485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.903482 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d924bb-eacb-4e53-a8f8-89fe595a9d69-bundle\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:04.903485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:04.903492 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj8tw\" (UniqueName: \"kubernetes.io/projected/96d924bb-eacb-4e53-a8f8-89fe595a9d69-kube-api-access-dj8tw\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:05.462718 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:05.462680 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" Jul 29 14:08:05.463157 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:05.462673 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm" event={"ID":"96d924bb-eacb-4e53-a8f8-89fe595a9d69","Type":"ContainerDied","Data":"1b3ddf8d5018eabf19c8faa66346bd2d870ed362491f21929d14c908d808be82"} Jul 29 14:08:05.463157 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:05.462810 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b3ddf8d5018eabf19c8faa66346bd2d870ed362491f21929d14c908d808be82" Jul 29 14:08:05.465088 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:05.465064 2560 generic.go:356] "Generic (PLEG): container finished" podID="839a6e05-d66f-4844-a6b6-144db923302b" containerID="2e5e49bba3042c70f5e14a8ec2b76150b4a1b1553169e3e4d765b79851ad6f5f" exitCode=0 Jul 29 14:08:05.465221 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:05.465162 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" event={"ID":"839a6e05-d66f-4844-a6b6-144db923302b","Type":"ContainerDied","Data":"2e5e49bba3042c70f5e14a8ec2b76150b4a1b1553169e3e4d765b79851ad6f5f"} Jul 29 14:08:05.467052 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:05.467030 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" event={"ID":"fb0151f9-7a05-44e3-b38a-3d6d96b7363e","Type":"ContainerDied","Data":"0fde20e01e5bf518046e8bbefb71e147e580f674c5f356c2fe4a2cb48627676e"} Jul 29 14:08:05.467052 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:05.467050 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh" Jul 29 14:08:05.467164 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:05.467073 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fde20e01e5bf518046e8bbefb71e147e580f674c5f356c2fe4a2cb48627676e" Jul 29 14:08:06.606590 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.606562 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:08:06.718613 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.718571 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cwd2\" (UniqueName: \"kubernetes.io/projected/839a6e05-d66f-4844-a6b6-144db923302b-kube-api-access-7cwd2\") pod \"839a6e05-d66f-4844-a6b6-144db923302b\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " Jul 29 14:08:06.718766 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.718636 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-bundle\") pod \"839a6e05-d66f-4844-a6b6-144db923302b\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " Jul 29 14:08:06.718766 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.718687 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-util\") pod \"839a6e05-d66f-4844-a6b6-144db923302b\" (UID: \"839a6e05-d66f-4844-a6b6-144db923302b\") " Jul 29 14:08:06.719185 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.719155 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-bundle" (OuterVolumeSpecName: "bundle") pod "839a6e05-d66f-4844-a6b6-144db923302b" (UID: "839a6e05-d66f-4844-a6b6-144db923302b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:08:06.721042 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.721010 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839a6e05-d66f-4844-a6b6-144db923302b-kube-api-access-7cwd2" (OuterVolumeSpecName: "kube-api-access-7cwd2") pod "839a6e05-d66f-4844-a6b6-144db923302b" (UID: "839a6e05-d66f-4844-a6b6-144db923302b"). InnerVolumeSpecName "kube-api-access-7cwd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:08:06.723419 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.723377 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-util" (OuterVolumeSpecName: "util") pod "839a6e05-d66f-4844-a6b6-144db923302b" (UID: "839a6e05-d66f-4844-a6b6-144db923302b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:08:06.819866 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.819838 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-bundle\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:06.819866 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.819861 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/839a6e05-d66f-4844-a6b6-144db923302b-util\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:06.819866 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:06.819871 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cwd2\" (UniqueName: \"kubernetes.io/projected/839a6e05-d66f-4844-a6b6-144db923302b-kube-api-access-7cwd2\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:07.479281 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:07.479249 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" event={"ID":"839a6e05-d66f-4844-a6b6-144db923302b","Type":"ContainerDied","Data":"a6805946e87fff0b2a8fddb2974d5fb61b4c9fd97e1f176254bbb5e6d34e4a70"} Jul 29 14:08:07.479484 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:07.479308 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6805946e87fff0b2a8fddb2974d5fb61b4c9fd97e1f176254bbb5e6d34e4a70" Jul 29 14:08:07.479484 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:07.479258 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx" Jul 29 14:08:14.974091 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974052 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt"] Jul 29 14:08:14.974466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974419 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerName="pull" Jul 29 14:08:14.974466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974430 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerName="pull" Jul 29 14:08:14.974466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974439 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839a6e05-d66f-4844-a6b6-144db923302b" containerName="extract" Jul 29 14:08:14.974466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974444 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="839a6e05-d66f-4844-a6b6-144db923302b" containerName="extract" Jul 29 14:08:14.974466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974451 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerName="extract" Jul 29 14:08:14.974466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974456 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerName="extract" Jul 29 14:08:14.974466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974466 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerName="extract" Jul 29 14:08:14.974466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974471 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerName="extract" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974479 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerName="pull" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974484 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerName="pull" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974490 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerName="util" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974495 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerName="util" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974504 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerName="util" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974509 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerName="util" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974517 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839a6e05-d66f-4844-a6b6-144db923302b" containerName="util" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974522 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="839a6e05-d66f-4844-a6b6-144db923302b" containerName="util" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974526 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839a6e05-d66f-4844-a6b6-144db923302b" containerName="pull" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974532 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="839a6e05-d66f-4844-a6b6-144db923302b" containerName="pull" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974556 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerName="pull" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974560 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerName="pull" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974566 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerName="extract" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974571 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerName="extract" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974578 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerName="util" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974583 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerName="util" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974638 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3301ee40-54e0-47b9-94f1-357d7763fb4f" containerName="extract" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974646 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0151f9-7a05-44e3-b38a-3d6d96b7363e" containerName="extract" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974652 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="839a6e05-d66f-4844-a6b6-144db923302b" containerName="extract" Jul 29 14:08:14.974733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.974660 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d924bb-eacb-4e53-a8f8-89fe595a9d69" containerName="extract" Jul 29 14:08:14.979363 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.979344 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" Jul 29 14:08:14.981721 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.981701 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 29 14:08:14.981831 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.981743 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6jvgl\"" Jul 29 14:08:14.988476 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:14.988453 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt"] Jul 29 14:08:15.088231 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:15.088188 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpslf\" (UniqueName: \"kubernetes.io/projected/c5017b6c-3682-44ab-9e76-8d3daea4c58b-kube-api-access-qpslf\") pod \"dns-operator-controller-manager-864dd78c4c-gbqkt\" (UID: \"c5017b6c-3682-44ab-9e76-8d3daea4c58b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" Jul 29 14:08:15.189503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:15.189471 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpslf\" (UniqueName: \"kubernetes.io/projected/c5017b6c-3682-44ab-9e76-8d3daea4c58b-kube-api-access-qpslf\") pod \"dns-operator-controller-manager-864dd78c4c-gbqkt\" (UID: \"c5017b6c-3682-44ab-9e76-8d3daea4c58b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" Jul 29 14:08:15.206370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:15.206334 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpslf\" (UniqueName: \"kubernetes.io/projected/c5017b6c-3682-44ab-9e76-8d3daea4c58b-kube-api-access-qpslf\") pod \"dns-operator-controller-manager-864dd78c4c-gbqkt\" (UID: \"c5017b6c-3682-44ab-9e76-8d3daea4c58b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" Jul 29 14:08:15.291379 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:15.291306 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" Jul 29 14:08:15.433376 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:15.433355 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt"] Jul 29 14:08:15.435480 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:15.435449 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5017b6c_3682_44ab_9e76_8d3daea4c58b.slice/crio-a42b11324556693bc40320512852ed6815522b6bedd02d14b67818bf3168a5f2 WatchSource:0}: Error finding container a42b11324556693bc40320512852ed6815522b6bedd02d14b67818bf3168a5f2: Status 404 returned error can't find the container with id a42b11324556693bc40320512852ed6815522b6bedd02d14b67818bf3168a5f2 Jul 29 14:08:15.530102 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:15.530064 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" event={"ID":"c5017b6c-3682-44ab-9e76-8d3daea4c58b","Type":"ContainerStarted","Data":"a42b11324556693bc40320512852ed6815522b6bedd02d14b67818bf3168a5f2"} Jul 29 14:08:17.187014 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.186974 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d45d947f5-hz9nk"] Jul 29 14:08:17.191206 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.191186 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.193778 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.193758 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 14:08:17.193923 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.193761 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 14:08:17.193923 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.193898 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 14:08:17.194576 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.194561 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 14:08:17.194747 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.194727 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 14:08:17.194807 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.194786 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 14:08:17.194858 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.194848 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g27sc\"" Jul 29 14:08:17.198409 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.198386 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 14:08:17.201292 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.201255 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 14:08:17.202141 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.202123 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d45d947f5-hz9nk"] Jul 29 14:08:17.309636 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.309605 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a26c1bb4-437c-43e0-b9b4-98be46a85954-console-oauth-config\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.309791 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.309647 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a26c1bb4-437c-43e0-b9b4-98be46a85954-console-serving-cert\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.309791 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.309728 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-service-ca\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.309791 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.309780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-oauth-serving-cert\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.309903 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.309822 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsggj\" (UniqueName: \"kubernetes.io/projected/a26c1bb4-437c-43e0-b9b4-98be46a85954-kube-api-access-vsggj\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.309903 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.309888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-console-config\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.309963 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.309921 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-trusted-ca-bundle\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.411250 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.411218 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsggj\" (UniqueName: \"kubernetes.io/projected/a26c1bb4-437c-43e0-b9b4-98be46a85954-kube-api-access-vsggj\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.411448 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.411303 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-console-config\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.411448 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.411342 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-trusted-ca-bundle\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.411448 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.411370 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a26c1bb4-437c-43e0-b9b4-98be46a85954-console-oauth-config\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.411448 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.411406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a26c1bb4-437c-43e0-b9b4-98be46a85954-console-serving-cert\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.411448 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.411436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-service-ca\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.411947 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.411483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-oauth-serving-cert\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.412168 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.412117 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-console-config\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.412283 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.412259 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-service-ca\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.412511 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.412259 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-oauth-serving-cert\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.412511 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.412365 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a26c1bb4-437c-43e0-b9b4-98be46a85954-trusted-ca-bundle\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.414330 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.414308 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a26c1bb4-437c-43e0-b9b4-98be46a85954-console-oauth-config\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.414573 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.414555 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a26c1bb4-437c-43e0-b9b4-98be46a85954-console-serving-cert\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.420184 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.420159 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsggj\" (UniqueName: \"kubernetes.io/projected/a26c1bb4-437c-43e0-b9b4-98be46a85954-kube-api-access-vsggj\") pod \"console-7d45d947f5-hz9nk\" (UID: \"a26c1bb4-437c-43e0-b9b4-98be46a85954\") " pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.503744 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.503646 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:17.783282 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:17.783260 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d45d947f5-hz9nk"] Jul 29 14:08:17.785464 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:17.785440 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26c1bb4_437c_43e0_b9b4_98be46a85954.slice/crio-3f370201e113912f231cae89e72dcdeea520305ff589b06145a8d584af17f7c8 WatchSource:0}: Error finding container 3f370201e113912f231cae89e72dcdeea520305ff589b06145a8d584af17f7c8: Status 404 returned error can't find the container with id 3f370201e113912f231cae89e72dcdeea520305ff589b06145a8d584af17f7c8 Jul 29 14:08:18.547807 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:18.547779 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d45d947f5-hz9nk" event={"ID":"a26c1bb4-437c-43e0-b9b4-98be46a85954","Type":"ContainerStarted","Data":"3f370201e113912f231cae89e72dcdeea520305ff589b06145a8d584af17f7c8"} Jul 29 14:08:18.549277 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:18.549244 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" event={"ID":"c5017b6c-3682-44ab-9e76-8d3daea4c58b","Type":"ContainerStarted","Data":"4ac9864cf4e2dd9630a67f9f5b8f910ddd36ad22fdb0629a07fcae4217efd9df"} Jul 29 14:08:18.549419 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:18.549402 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" Jul 29 14:08:18.567854 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:18.567810 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" podStartSLOduration=2.300745432 podStartE2EDuration="4.567796652s" totalImagesPullingTime="2.26705122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:08:18.565911455 +0000 UTC m=+431.767412626" watchObservedRunningTime="2026-07-29 14:08:18.567796652 +0000 UTC m=+431.769297824" Jul 29 14:08:19.023438 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.023408 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb"] Jul 29 14:08:19.029673 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.029656 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" Jul 29 14:08:19.032170 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.032149 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pd9qr\"" Jul 29 14:08:19.035938 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.035915 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb"] Jul 29 14:08:19.126018 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.125985 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkm75\" (UniqueName: \"kubernetes.io/projected/27b7073f-03b6-41aa-8f19-6d5b71ef82bf-kube-api-access-pkm75\") pod \"authorino-operator-75cc8d5cb7-rkqhb\" (UID: \"27b7073f-03b6-41aa-8f19-6d5b71ef82bf\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" Jul 29 14:08:19.226990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.226959 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkm75\" (UniqueName: \"kubernetes.io/projected/27b7073f-03b6-41aa-8f19-6d5b71ef82bf-kube-api-access-pkm75\") pod \"authorino-operator-75cc8d5cb7-rkqhb\" (UID: \"27b7073f-03b6-41aa-8f19-6d5b71ef82bf\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" Jul 29 14:08:19.236715 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.236685 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkm75\" (UniqueName: \"kubernetes.io/projected/27b7073f-03b6-41aa-8f19-6d5b71ef82bf-kube-api-access-pkm75\") pod \"authorino-operator-75cc8d5cb7-rkqhb\" (UID: \"27b7073f-03b6-41aa-8f19-6d5b71ef82bf\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" Jul 29 14:08:19.341026 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.340948 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" Jul 29 14:08:19.487908 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.487880 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb"] Jul 29 14:08:19.489302 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:19.489280 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b7073f_03b6_41aa_8f19_6d5b71ef82bf.slice/crio-e7df066a6ba96b223653bcdf57c46023aa030dfda1f9eedbece9f8ad29b49780 WatchSource:0}: Error finding container e7df066a6ba96b223653bcdf57c46023aa030dfda1f9eedbece9f8ad29b49780: Status 404 returned error can't find the container with id e7df066a6ba96b223653bcdf57c46023aa030dfda1f9eedbece9f8ad29b49780 Jul 29 14:08:19.554979 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:19.554949 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" event={"ID":"27b7073f-03b6-41aa-8f19-6d5b71ef82bf","Type":"ContainerStarted","Data":"e7df066a6ba96b223653bcdf57c46023aa030dfda1f9eedbece9f8ad29b49780"} Jul 29 14:08:22.201199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.201162 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247"] Jul 29 14:08:22.204861 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.204842 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:22.209085 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.209067 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-skfhd\"" Jul 29 14:08:22.225561 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.225526 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247"] Jul 29 14:08:22.355402 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.355367 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhgg\" (UniqueName: \"kubernetes.io/projected/b7951587-30a6-468e-934d-41735372c8fe-kube-api-access-glhgg\") pod \"limitador-operator-controller-manager-859d7b78bd-4m247\" (UID: \"b7951587-30a6-468e-934d-41735372c8fe\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:22.456967 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.456886 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhgg\" (UniqueName: \"kubernetes.io/projected/b7951587-30a6-468e-934d-41735372c8fe-kube-api-access-glhgg\") pod \"limitador-operator-controller-manager-859d7b78bd-4m247\" (UID: \"b7951587-30a6-468e-934d-41735372c8fe\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:22.468791 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.468768 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhgg\" (UniqueName: \"kubernetes.io/projected/b7951587-30a6-468e-934d-41735372c8fe-kube-api-access-glhgg\") pod \"limitador-operator-controller-manager-859d7b78bd-4m247\" (UID: \"b7951587-30a6-468e-934d-41735372c8fe\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:22.514689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.514656 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:22.577724 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.577680 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" event={"ID":"27b7073f-03b6-41aa-8f19-6d5b71ef82bf","Type":"ContainerStarted","Data":"fce991d243b8f2388bb7aa0b82fce06170e9ad4f60b11abd57953feb6259cd4c"} Jul 29 14:08:22.577935 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.577892 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" Jul 29 14:08:22.604129 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.602274 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" podStartSLOduration=2.439621892 podStartE2EDuration="4.60225114s" totalImagesPullingTime="2.162629248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:08:22.600890728 +0000 UTC m=+435.802391902" watchObservedRunningTime="2026-07-29 14:08:22.60225114 +0000 UTC m=+435.803752311" Jul 29 14:08:22.684848 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:22.684769 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247"] Jul 29 14:08:22.685601 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:22.685574 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7951587_30a6_468e_934d_41735372c8fe.slice/crio-ee7c56efcf6e529c59f03d035b7d759262cddb5024f1cfd7e99684171168ca98 WatchSource:0}: Error finding container ee7c56efcf6e529c59f03d035b7d759262cddb5024f1cfd7e99684171168ca98: Status 404 returned error can't find the container with id ee7c56efcf6e529c59f03d035b7d759262cddb5024f1cfd7e99684171168ca98 Jul 29 14:08:23.587701 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:23.587665 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" event={"ID":"b7951587-30a6-468e-934d-41735372c8fe","Type":"ContainerStarted","Data":"ee7c56efcf6e529c59f03d035b7d759262cddb5024f1cfd7e99684171168ca98"} Jul 29 14:08:25.602804 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:25.602763 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" event={"ID":"b7951587-30a6-468e-934d-41735372c8fe","Type":"ContainerStarted","Data":"7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb"} Jul 29 14:08:25.603262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:25.602839 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:25.604383 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:25.604350 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d45d947f5-hz9nk" event={"ID":"a26c1bb4-437c-43e0-b9b4-98be46a85954","Type":"ContainerStarted","Data":"83913a6777bc9bcb14d8bd0ea1276e26d9a8127fa105019168885b080ec1d88d"} Jul 29 14:08:25.621601 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:25.621528 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" podStartSLOduration=0.861379828 podStartE2EDuration="3.621513189s" totalImagesPullingTime="2.760133361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:08:25.619203287 +0000 UTC m=+438.820704460" watchObservedRunningTime="2026-07-29 14:08:25.621513189 +0000 UTC m=+438.823014361" Jul 29 14:08:25.641138 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:25.641080 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d45d947f5-hz9nk" podStartSLOduration=0.977157036 podStartE2EDuration="8.641062363s" totalImagesPullingTime="7.663905327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:08:25.639733869 +0000 UTC m=+438.841235042" watchObservedRunningTime="2026-07-29 14:08:25.641062363 +0000 UTC m=+438.842563539" Jul 29 14:08:26.981103 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:26.981066 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5"] Jul 29 14:08:26.984855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:26.984836 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:26.987658 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:26.987634 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2nwr\"" Jul 29 14:08:26.998497 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:26.998465 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5"] Jul 29 14:08:27.101503 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.101466 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60d4d5fe-bac7-44bc-94ab-09fa4830e763-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" (UID: \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:27.101710 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.101515 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8z7v\" (UniqueName: \"kubernetes.io/projected/60d4d5fe-bac7-44bc-94ab-09fa4830e763-kube-api-access-w8z7v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" (UID: \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:27.202523 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.202489 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60d4d5fe-bac7-44bc-94ab-09fa4830e763-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" (UID: \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:27.202731 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.202533 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8z7v\" (UniqueName: \"kubernetes.io/projected/60d4d5fe-bac7-44bc-94ab-09fa4830e763-kube-api-access-w8z7v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" (UID: \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:27.202910 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.202889 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60d4d5fe-bac7-44bc-94ab-09fa4830e763-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" (UID: \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:27.212042 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.212011 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8z7v\" (UniqueName: \"kubernetes.io/projected/60d4d5fe-bac7-44bc-94ab-09fa4830e763-kube-api-access-w8z7v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" (UID: \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:27.295389 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.295287 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:27.436212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.436188 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5"] Jul 29 14:08:27.438328 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:27.438288 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d4d5fe_bac7_44bc_94ab_09fa4830e763.slice/crio-eab11163e4825f236b05b6f29e39d04c1d04d29b47f1bd9a7d813737fcc1960e WatchSource:0}: Error finding container eab11163e4825f236b05b6f29e39d04c1d04d29b47f1bd9a7d813737fcc1960e: Status 404 returned error can't find the container with id eab11163e4825f236b05b6f29e39d04c1d04d29b47f1bd9a7d813737fcc1960e Jul 29 14:08:27.504342 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.504301 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:27.504517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.504405 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:27.509799 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.509769 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:27.616784 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.616691 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" event={"ID":"60d4d5fe-bac7-44bc-94ab-09fa4830e763","Type":"ContainerStarted","Data":"eab11163e4825f236b05b6f29e39d04c1d04d29b47f1bd9a7d813737fcc1960e"} Jul 29 14:08:27.621326 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:27.621301 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d45d947f5-hz9nk" Jul 29 14:08:29.558279 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:29.558193 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gbqkt" Jul 29 14:08:33.591161 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:33.591132 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rkqhb" Jul 29 14:08:33.660303 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:33.660267 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" event={"ID":"60d4d5fe-bac7-44bc-94ab-09fa4830e763","Type":"ContainerStarted","Data":"ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259"} Jul 29 14:08:33.660487 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:33.660331 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:33.686810 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:33.686757 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" podStartSLOduration=1.9867729650000001 podStartE2EDuration="7.686740711s" totalImagesPullingTime="5.699967746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:08:33.679838799 +0000 UTC m=+446.881339971" watchObservedRunningTime="2026-07-29 14:08:33.686740711 +0000 UTC m=+446.888241883" Jul 29 14:08:35.750353 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.750318 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6"] Jul 29 14:08:35.755731 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.755697 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:35.757990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.757969 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 29 14:08:35.758119 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.758010 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fcttf\"" Jul 29 14:08:35.758254 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.758239 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 29 14:08:35.763963 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.763941 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6"] Jul 29 14:08:35.769726 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.769700 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2vd6\" (UniqueName: \"kubernetes.io/projected/cd43c72b-7401-4f5e-b3a3-de3dae69014d-kube-api-access-l2vd6\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:35.769844 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.769778 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd43c72b-7401-4f5e-b3a3-de3dae69014d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:35.769844 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.769804 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd43c72b-7401-4f5e-b3a3-de3dae69014d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:35.871028 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.870992 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2vd6\" (UniqueName: \"kubernetes.io/projected/cd43c72b-7401-4f5e-b3a3-de3dae69014d-kube-api-access-l2vd6\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:35.871194 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.871060 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd43c72b-7401-4f5e-b3a3-de3dae69014d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:35.871194 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.871088 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd43c72b-7401-4f5e-b3a3-de3dae69014d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:35.871269 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:35.871203 2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 29 14:08:35.871303 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:35.871271 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd43c72b-7401-4f5e-b3a3-de3dae69014d-plugin-serving-cert podName:cd43c72b-7401-4f5e-b3a3-de3dae69014d nodeName:}" failed. No retries permitted until 2026-07-29 14:08:36.371248945 +0000 UTC m=+449.572750096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/cd43c72b-7401-4f5e-b3a3-de3dae69014d-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-bg5c6" (UID: "cd43c72b-7401-4f5e-b3a3-de3dae69014d") : secret "plugin-serving-cert" not found Jul 29 14:08:35.871814 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.871794 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd43c72b-7401-4f5e-b3a3-de3dae69014d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:35.880092 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:35.880071 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2vd6\" (UniqueName: \"kubernetes.io/projected/cd43c72b-7401-4f5e-b3a3-de3dae69014d-kube-api-access-l2vd6\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:36.375773 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:36.375736 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd43c72b-7401-4f5e-b3a3-de3dae69014d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:36.378354 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:36.378330 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd43c72b-7401-4f5e-b3a3-de3dae69014d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bg5c6\" (UID: \"cd43c72b-7401-4f5e-b3a3-de3dae69014d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:36.612228 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:36.612195 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:36.665821 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:36.665740 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" Jul 29 14:08:36.805469 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:36.805440 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6"] Jul 29 14:08:36.807853 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:36.807827 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd43c72b_7401_4f5e_b3a3_de3dae69014d.slice/crio-b89f3b4052eead4247844420d7ff4fa17df75076c45889f5b734fe3cb73e0de9 WatchSource:0}: Error finding container b89f3b4052eead4247844420d7ff4fa17df75076c45889f5b734fe3cb73e0de9: Status 404 returned error can't find the container with id b89f3b4052eead4247844420d7ff4fa17df75076c45889f5b734fe3cb73e0de9 Jul 29 14:08:37.685259 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:37.685220 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" event={"ID":"cd43c72b-7401-4f5e-b3a3-de3dae69014d","Type":"ContainerStarted","Data":"b89f3b4052eead4247844420d7ff4fa17df75076c45889f5b734fe3cb73e0de9"} Jul 29 14:08:44.668531 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:44.668502 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:45.610331 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.610283 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp"] Jul 29 14:08:45.621340 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.621289 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:08:45.624093 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.624063 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp"] Jul 29 14:08:45.669589 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.669529 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwcb\" (UniqueName: \"kubernetes.io/projected/3eea8ed3-9375-45dd-9560-ed318c45ad40-kube-api-access-5lwcb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" (UID: \"3eea8ed3-9375-45dd-9560-ed318c45ad40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:08:45.670009 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.669644 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3eea8ed3-9375-45dd-9560-ed318c45ad40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" (UID: \"3eea8ed3-9375-45dd-9560-ed318c45ad40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:08:45.771131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.771089 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3eea8ed3-9375-45dd-9560-ed318c45ad40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" (UID: \"3eea8ed3-9375-45dd-9560-ed318c45ad40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:08:45.771327 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.771174 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwcb\" (UniqueName: \"kubernetes.io/projected/3eea8ed3-9375-45dd-9560-ed318c45ad40-kube-api-access-5lwcb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" (UID: \"3eea8ed3-9375-45dd-9560-ed318c45ad40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:08:45.771500 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.771477 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3eea8ed3-9375-45dd-9560-ed318c45ad40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" (UID: \"3eea8ed3-9375-45dd-9560-ed318c45ad40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:08:45.780678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.780652 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwcb\" (UniqueName: \"kubernetes.io/projected/3eea8ed3-9375-45dd-9560-ed318c45ad40-kube-api-access-5lwcb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" (UID: \"3eea8ed3-9375-45dd-9560-ed318c45ad40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:08:45.936505 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:45.936469 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:08:46.378428 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.378322 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp"] Jul 29 14:08:46.382030 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.381995 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp"] Jul 29 14:08:46.394863 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.394831 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5"] Jul 29 14:08:46.395140 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.395087 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" containerName="manager" containerID="cri-o://ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259" gracePeriod=2 Jul 29 14:08:46.398412 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.397940 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b"] Jul 29 14:08:46.404689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.403595 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:08:46.410792 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.410756 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5"] Jul 29 14:08:46.418699 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:46.418659 2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"60d4d5fe-bac7-44bc-94ab-09fa4830e763\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\": pods \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:46.419907 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.419886 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b"] Jul 29 14:08:46.426504 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.426455 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247"] Jul 29 14:08:46.426967 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.426921 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" podUID="b7951587-30a6-468e-934d-41735372c8fe" containerName="manager" containerID="cri-o://7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb" gracePeriod=2 Jul 29 14:08:46.430687 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.430212 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk"] Jul 29 14:08:46.430829 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.430810 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" containerName="manager" Jul 29 14:08:46.430949 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.430829 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" containerName="manager" Jul 29 14:08:46.431036 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.430964 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" containerName="manager" Jul 29 14:08:46.436559 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.436513 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247"] Jul 29 14:08:46.437226 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.437202 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:08:46.444381 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:46.444169 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:46.448388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.448366 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk"] Jul 29 14:08:46.471805 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:46.471766 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:46.473749 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:46.473702 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4m247\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="b7951587-30a6-468e-934d-41735372c8fe" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:46.478282 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.478247 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a0d09-9fae-4306-a838-7132020fb567-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zz64b\" (UID: \"952a0d09-9fae-4306-a838-7132020fb567\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:08:46.478456 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.478304 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9nn5\" (UniqueName: \"kubernetes.io/projected/9889afa9-db76-496d-aa5d-a7e82c981a0f-kube-api-access-f9nn5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8v7fk\" (UID: \"9889afa9-db76-496d-aa5d-a7e82c981a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:08:46.478456 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.478449 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dt6k\" (UniqueName: \"kubernetes.io/projected/952a0d09-9fae-4306-a838-7132020fb567-kube-api-access-6dt6k\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zz64b\" (UID: \"952a0d09-9fae-4306-a838-7132020fb567\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:08:46.478650 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.478484 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9889afa9-db76-496d-aa5d-a7e82c981a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8v7fk\" (UID: \"9889afa9-db76-496d-aa5d-a7e82c981a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:08:46.579951 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.579907 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dt6k\" (UniqueName: \"kubernetes.io/projected/952a0d09-9fae-4306-a838-7132020fb567-kube-api-access-6dt6k\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zz64b\" (UID: \"952a0d09-9fae-4306-a838-7132020fb567\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:08:46.579951 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.579964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9889afa9-db76-496d-aa5d-a7e82c981a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8v7fk\" (UID: \"9889afa9-db76-496d-aa5d-a7e82c981a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:08:46.580217 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.580076 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a0d09-9fae-4306-a838-7132020fb567-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zz64b\" (UID: \"952a0d09-9fae-4306-a838-7132020fb567\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:08:46.580217 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.580110 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9nn5\" (UniqueName: \"kubernetes.io/projected/9889afa9-db76-496d-aa5d-a7e82c981a0f-kube-api-access-f9nn5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8v7fk\" (UID: \"9889afa9-db76-496d-aa5d-a7e82c981a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:08:46.580432 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.580403 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9889afa9-db76-496d-aa5d-a7e82c981a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8v7fk\" (UID: \"9889afa9-db76-496d-aa5d-a7e82c981a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:08:46.580598 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.580565 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a0d09-9fae-4306-a838-7132020fb567-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zz64b\" (UID: \"952a0d09-9fae-4306-a838-7132020fb567\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:08:46.591655 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.591620 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9nn5\" (UniqueName: \"kubernetes.io/projected/9889afa9-db76-496d-aa5d-a7e82c981a0f-kube-api-access-f9nn5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8v7fk\" (UID: \"9889afa9-db76-496d-aa5d-a7e82c981a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:08:46.591655 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.591637 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dt6k\" (UniqueName: \"kubernetes.io/projected/952a0d09-9fae-4306-a838-7132020fb567-kube-api-access-6dt6k\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zz64b\" (UID: \"952a0d09-9fae-4306-a838-7132020fb567\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:08:46.810712 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.810676 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:08:46.817855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:46.817828 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:08:47.380401 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:47.380360 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:47.382074 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:47.382048 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4m247\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="b7951587-30a6-468e-934d-41735372c8fe" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:47.611700 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:47.611612 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" podUID="b7951587-30a6-468e-934d-41735372c8fe" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.43:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 14:08:58.780022 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.779989 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:58.782068 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:58.782031 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4m247\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="b7951587-30a6-468e-934d-41735372c8fe" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:58.791329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.790246 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b"] Jul 29 14:08:58.793885 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:58.793861 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952a0d09_9fae_4306_a838_7132020fb567.slice/crio-77053433ef442a66bb8f31183e3cf2d55daab520848bfd266a645717d9ab0c6a WatchSource:0}: Error finding container 77053433ef442a66bb8f31183e3cf2d55daab520848bfd266a645717d9ab0c6a: Status 404 returned error can't find the container with id 77053433ef442a66bb8f31183e3cf2d55daab520848bfd266a645717d9ab0c6a Jul 29 14:08:58.794822 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.794802 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:58.798050 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:58.798021 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4m247\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="b7951587-30a6-468e-934d-41735372c8fe" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:58.800195 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:58.800159 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:58.846345 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.846323 2560 generic.go:356] "Generic (PLEG): container finished" podID="b7951587-30a6-468e-934d-41735372c8fe" containerID="7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb" exitCode=0 Jul 29 14:08:58.846432 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.846406 2560 scope.go:117] "RemoveContainer" containerID="7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb" Jul 29 14:08:58.846624 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.846531 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:58.848451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.848422 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" event={"ID":"952a0d09-9fae-4306-a838-7132020fb567","Type":"ContainerStarted","Data":"77053433ef442a66bb8f31183e3cf2d55daab520848bfd266a645717d9ab0c6a"} Jul 29 14:08:58.850219 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:58.849927 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:58.851407 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.851341 2560 generic.go:356] "Generic (PLEG): container finished" podID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" containerID="ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259" exitCode=0 Jul 29 14:08:58.851407 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.851389 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:58.851964 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:58.851943 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4m247\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="b7951587-30a6-468e-934d-41735372c8fe" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:58.854334 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:58.854006 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:58.857871 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:58.857838 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4m247\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="b7951587-30a6-468e-934d-41735372c8fe" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:58.861155 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.861133 2560 scope.go:117] "RemoveContainer" containerID="7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb" Jul 29 14:08:58.862195 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:58.861856 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb\": container with ID starting with 7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb not found: ID does not exist" containerID="7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb" Jul 29 14:08:58.862195 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.861897 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb"} err="failed to get container status \"7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb\": rpc error: code = NotFound desc = could not find container \"7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb\": container with ID starting with 7983f7c55c937fd1292168e82f18025a1b8a1cded0e036d5bcf969543045bbeb not found: ID does not exist" Jul 29 14:08:58.862195 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.861920 2560 scope.go:117] "RemoveContainer" containerID="ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259" Jul 29 14:08:58.873603 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.873496 2560 scope.go:117] "RemoveContainer" containerID="ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259" Jul 29 14:08:58.874015 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:58.873987 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259\": container with ID starting with ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259 not found: ID does not exist" containerID="ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259" Jul 29 14:08:58.874120 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.874024 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259"} err="failed to get container status \"ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259\": rpc error: code = NotFound desc = could not find container \"ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259\": container with ID starting with ba869db71878b4dfc3eb621019f1a3f38d79733026185204ce5e8b5adc18d259 not found: ID does not exist" Jul 29 14:08:58.896494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.896471 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8z7v\" (UniqueName: \"kubernetes.io/projected/60d4d5fe-bac7-44bc-94ab-09fa4830e763-kube-api-access-w8z7v\") pod \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\" (UID: \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\") " Jul 29 14:08:58.896657 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.896510 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glhgg\" (UniqueName: \"kubernetes.io/projected/b7951587-30a6-468e-934d-41735372c8fe-kube-api-access-glhgg\") pod \"b7951587-30a6-468e-934d-41735372c8fe\" (UID: \"b7951587-30a6-468e-934d-41735372c8fe\") " Jul 29 14:08:58.896657 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.896607 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60d4d5fe-bac7-44bc-94ab-09fa4830e763-extensions-socket-volume\") pod \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\" (UID: \"60d4d5fe-bac7-44bc-94ab-09fa4830e763\") " Jul 29 14:08:58.897035 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.897001 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d4d5fe-bac7-44bc-94ab-09fa4830e763-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "60d4d5fe-bac7-44bc-94ab-09fa4830e763" (UID: "60d4d5fe-bac7-44bc-94ab-09fa4830e763"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:08:58.898508 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.898479 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d4d5fe-bac7-44bc-94ab-09fa4830e763-kube-api-access-w8z7v" (OuterVolumeSpecName: "kube-api-access-w8z7v") pod "60d4d5fe-bac7-44bc-94ab-09fa4830e763" (UID: "60d4d5fe-bac7-44bc-94ab-09fa4830e763"). InnerVolumeSpecName "kube-api-access-w8z7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:08:58.899107 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.899086 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7951587-30a6-468e-934d-41735372c8fe-kube-api-access-glhgg" (OuterVolumeSpecName: "kube-api-access-glhgg") pod "b7951587-30a6-468e-934d-41735372c8fe" (UID: "b7951587-30a6-468e-934d-41735372c8fe"). InnerVolumeSpecName "kube-api-access-glhgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:08:58.919524 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:08:58.919501 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9889afa9_db76_496d_aa5d_a7e82c981a0f.slice/crio-b9eae24fe9e0f151095d5f2f523d0ead94c9888195c00b1f4b39cc45320c0b22 WatchSource:0}: Error finding container b9eae24fe9e0f151095d5f2f523d0ead94c9888195c00b1f4b39cc45320c0b22: Status 404 returned error can't find the container with id b9eae24fe9e0f151095d5f2f523d0ead94c9888195c00b1f4b39cc45320c0b22 Jul 29 14:08:58.920387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.920365 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk"] Jul 29 14:08:58.997440 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.997414 2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60d4d5fe-bac7-44bc-94ab-09fa4830e763-extensions-socket-volume\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:58.997440 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.997438 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8z7v\" (UniqueName: \"kubernetes.io/projected/60d4d5fe-bac7-44bc-94ab-09fa4830e763-kube-api-access-w8z7v\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:58.997691 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:58.997447 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhgg\" (UniqueName: \"kubernetes.io/projected/b7951587-30a6-468e-934d-41735372c8fe-kube-api-access-glhgg\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:08:59.157014 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:59.156983 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4m247\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="b7951587-30a6-468e-934d-41735372c8fe" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:59.158718 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:59.158693 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:59.161666 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:59.161641 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-p7qv5\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p7qv5" Jul 29 14:08:59.163148 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:08:59.163124 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4m247\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="b7951587-30a6-468e-934d-41735372c8fe" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4m247" Jul 29 14:08:59.382195 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.382164 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d4d5fe-bac7-44bc-94ab-09fa4830e763" path="/var/lib/kubelet/pods/60d4d5fe-bac7-44bc-94ab-09fa4830e763/volumes" Jul 29 14:08:59.382520 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.382506 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7951587-30a6-468e-934d-41735372c8fe" path="/var/lib/kubelet/pods/b7951587-30a6-468e-934d-41735372c8fe/volumes" Jul 29 14:08:59.859039 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.859006 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" event={"ID":"cd43c72b-7401-4f5e-b3a3-de3dae69014d","Type":"ContainerStarted","Data":"1460e72e05e67ca2027dedc0f46a2a211588389380ed9687b34cd7a1f8d6c3d5"} Jul 29 14:08:59.860725 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.860663 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" event={"ID":"9889afa9-db76-496d-aa5d-a7e82c981a0f","Type":"ContainerStarted","Data":"ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9"} Jul 29 14:08:59.860725 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.860706 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" event={"ID":"9889afa9-db76-496d-aa5d-a7e82c981a0f","Type":"ContainerStarted","Data":"b9eae24fe9e0f151095d5f2f523d0ead94c9888195c00b1f4b39cc45320c0b22"} Jul 29 14:08:59.860882 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.860747 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:08:59.863113 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.863095 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" event={"ID":"952a0d09-9fae-4306-a838-7132020fb567","Type":"ContainerStarted","Data":"f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5"} Jul 29 14:08:59.863245 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.863210 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:08:59.876173 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.876137 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bg5c6" podStartSLOduration=2.892234327 podStartE2EDuration="24.876120088s" totalImagesPullingTime="21.983885761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:08:59.875439118 +0000 UTC m=+473.076940291" watchObservedRunningTime="2026-07-29 14:08:59.876120088 +0000 UTC m=+473.077621264" Jul 29 14:08:59.898925 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.898864 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" podStartSLOduration=13.898841869 podStartE2EDuration="13.898841869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:08:59.894950297 +0000 UTC m=+473.096451470" watchObservedRunningTime="2026-07-29 14:08:59.898841869 +0000 UTC m=+473.100343042" Jul 29 14:08:59.914360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:08:59.914316 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" podStartSLOduration=13.914302953 podStartE2EDuration="13.914302953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:08:59.912903813 +0000 UTC m=+473.114404986" watchObservedRunningTime="2026-07-29 14:08:59.914302953 +0000 UTC m=+473.115804125" Jul 29 14:09:01.292789 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:09:01.292748 2560 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-7w4fp_kuadrant-system_3eea8ed3-9375-45dd-9560-ed318c45ad40_0(b19e9d6d14c4e1249ca968977b4fc3bce0714517d3225509a46763832f95d895): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-7w4fp to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 29 14:09:01.293067 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:09:01.292816 2560 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-7w4fp_kuadrant-system_3eea8ed3-9375-45dd-9560-ed318c45ad40_0(b19e9d6d14c4e1249ca968977b4fc3bce0714517d3225509a46763832f95d895): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-7w4fp to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:09:01.873419 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:01.873380 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:09:01.875590 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:09:01.875558 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="3eea8ed3-9375-45dd-9560-ed318c45ad40" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:09:01.880396 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:01.880378 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:09:01.882373 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:09:01.882345 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="3eea8ed3-9375-45dd-9560-ed318c45ad40" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:09:02.024809 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:02.024776 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lwcb\" (UniqueName: \"kubernetes.io/projected/3eea8ed3-9375-45dd-9560-ed318c45ad40-kube-api-access-5lwcb\") pod \"3eea8ed3-9375-45dd-9560-ed318c45ad40\" (UID: \"3eea8ed3-9375-45dd-9560-ed318c45ad40\") " Jul 29 14:09:02.024957 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:02.024851 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3eea8ed3-9375-45dd-9560-ed318c45ad40-extensions-socket-volume\") pod \"3eea8ed3-9375-45dd-9560-ed318c45ad40\" (UID: \"3eea8ed3-9375-45dd-9560-ed318c45ad40\") " Jul 29 14:09:02.025147 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:02.025125 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eea8ed3-9375-45dd-9560-ed318c45ad40-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3eea8ed3-9375-45dd-9560-ed318c45ad40" (UID: "3eea8ed3-9375-45dd-9560-ed318c45ad40"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:09:02.026999 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:02.026975 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eea8ed3-9375-45dd-9560-ed318c45ad40-kube-api-access-5lwcb" (OuterVolumeSpecName: "kube-api-access-5lwcb") pod "3eea8ed3-9375-45dd-9560-ed318c45ad40" (UID: "3eea8ed3-9375-45dd-9560-ed318c45ad40"). InnerVolumeSpecName "kube-api-access-5lwcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:09:02.126132 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:02.126055 2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3eea8ed3-9375-45dd-9560-ed318c45ad40-extensions-socket-volume\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:09:02.126132 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:02.126083 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lwcb\" (UniqueName: \"kubernetes.io/projected/3eea8ed3-9375-45dd-9560-ed318c45ad40-kube-api-access-5lwcb\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:09:02.878470 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:02.878440 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:09:02.880639 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:09:02.880603 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="3eea8ed3-9375-45dd-9560-ed318c45ad40" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:09:02.886288 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:09:02.886262 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7w4fp\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="3eea8ed3-9375-45dd-9560-ed318c45ad40" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7w4fp" Jul 29 14:09:03.380444 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:03.380416 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eea8ed3-9375-45dd-9560-ed318c45ad40" path="/var/lib/kubelet/pods/3eea8ed3-9375-45dd-9560-ed318c45ad40/volumes" Jul 29 14:09:10.870910 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:10.870880 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:09:10.871373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:10.871054 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:09:10.929101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:10.929071 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk"] Jul 29 14:09:10.929271 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:10.929243 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" podUID="9889afa9-db76-496d-aa5d-a7e82c981a0f" containerName="manager" containerID="cri-o://ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9" gracePeriod=10 Jul 29 14:09:11.157152 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.157074 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn"] Jul 29 14:09:11.157694 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.157675 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7951587-30a6-468e-934d-41735372c8fe" containerName="manager" Jul 29 14:09:11.157751 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.157698 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7951587-30a6-468e-934d-41735372c8fe" containerName="manager" Jul 29 14:09:11.157815 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.157804 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7951587-30a6-468e-934d-41735372c8fe" containerName="manager" Jul 29 14:09:11.159833 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.159811 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:11.172898 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.172874 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn"] Jul 29 14:09:11.192360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.192340 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:09:11.313135 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.313105 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9nn5\" (UniqueName: \"kubernetes.io/projected/9889afa9-db76-496d-aa5d-a7e82c981a0f-kube-api-access-f9nn5\") pod \"9889afa9-db76-496d-aa5d-a7e82c981a0f\" (UID: \"9889afa9-db76-496d-aa5d-a7e82c981a0f\") " Jul 29 14:09:11.313135 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.313139 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9889afa9-db76-496d-aa5d-a7e82c981a0f-extensions-socket-volume\") pod \"9889afa9-db76-496d-aa5d-a7e82c981a0f\" (UID: \"9889afa9-db76-496d-aa5d-a7e82c981a0f\") " Jul 29 14:09:11.313337 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.313306 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l6fwn\" (UID: \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:11.313414 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.313390 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn45q\" (UniqueName: \"kubernetes.io/projected/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-kube-api-access-vn45q\") pod \"kuadrant-operator-controller-manager-86878c769f-l6fwn\" (UID: \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:11.313602 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.313578 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9889afa9-db76-496d-aa5d-a7e82c981a0f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9889afa9-db76-496d-aa5d-a7e82c981a0f" (UID: "9889afa9-db76-496d-aa5d-a7e82c981a0f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:09:11.315265 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.315243 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9889afa9-db76-496d-aa5d-a7e82c981a0f-kube-api-access-f9nn5" (OuterVolumeSpecName: "kube-api-access-f9nn5") pod "9889afa9-db76-496d-aa5d-a7e82c981a0f" (UID: "9889afa9-db76-496d-aa5d-a7e82c981a0f"). InnerVolumeSpecName "kube-api-access-f9nn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:09:11.413990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.413922 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn45q\" (UniqueName: \"kubernetes.io/projected/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-kube-api-access-vn45q\") pod \"kuadrant-operator-controller-manager-86878c769f-l6fwn\" (UID: \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:11.413990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.413985 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l6fwn\" (UID: \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:11.414135 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.414061 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9nn5\" (UniqueName: \"kubernetes.io/projected/9889afa9-db76-496d-aa5d-a7e82c981a0f-kube-api-access-f9nn5\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:09:11.414135 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.414075 2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9889afa9-db76-496d-aa5d-a7e82c981a0f-extensions-socket-volume\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:09:11.414362 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.414345 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l6fwn\" (UID: \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:11.425666 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.425641 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn45q\" (UniqueName: \"kubernetes.io/projected/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-kube-api-access-vn45q\") pod \"kuadrant-operator-controller-manager-86878c769f-l6fwn\" (UID: \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:11.488458 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.488428 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:11.629510 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.629486 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn"] Jul 29 14:09:11.630876 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:09:11.630846 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9fe48f_8e6f_4db3_bfdd_67d9c20ff07f.slice/crio-1e9d7f6f74f4be030a12c4ed1e61ddf3474cf9450d2b3550b3be0a0314144e31 WatchSource:0}: Error finding container 1e9d7f6f74f4be030a12c4ed1e61ddf3474cf9450d2b3550b3be0a0314144e31: Status 404 returned error can't find the container with id 1e9d7f6f74f4be030a12c4ed1e61ddf3474cf9450d2b3550b3be0a0314144e31 Jul 29 14:09:11.933137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.933101 2560 generic.go:356] "Generic (PLEG): container finished" podID="9889afa9-db76-496d-aa5d-a7e82c981a0f" containerID="ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9" exitCode=0 Jul 29 14:09:11.933623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.933154 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" Jul 29 14:09:11.933623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.933186 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" event={"ID":"9889afa9-db76-496d-aa5d-a7e82c981a0f","Type":"ContainerDied","Data":"ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9"} Jul 29 14:09:11.933623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.933282 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk" event={"ID":"9889afa9-db76-496d-aa5d-a7e82c981a0f","Type":"ContainerDied","Data":"b9eae24fe9e0f151095d5f2f523d0ead94c9888195c00b1f4b39cc45320c0b22"} Jul 29 14:09:11.933623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.933313 2560 scope.go:117] "RemoveContainer" containerID="ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9" Jul 29 14:09:11.935522 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.935485 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" event={"ID":"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f","Type":"ContainerStarted","Data":"b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464"} Jul 29 14:09:11.935725 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.935699 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" event={"ID":"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f","Type":"ContainerStarted","Data":"1e9d7f6f74f4be030a12c4ed1e61ddf3474cf9450d2b3550b3be0a0314144e31"} Jul 29 14:09:11.935838 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.935734 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:11.942804 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.942785 2560 scope.go:117] "RemoveContainer" containerID="ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9" Jul 29 14:09:11.943038 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:09:11.943020 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9\": container with ID starting with ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9 not found: ID does not exist" containerID="ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9" Jul 29 14:09:11.943107 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.943050 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9"} err="failed to get container status \"ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9\": rpc error: code = NotFound desc = could not find container \"ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9\": container with ID starting with ec9c11f0517e55ccb73a3c2c7c3f79d1aa570a6ed7d3defce57dee91097fc4c9 not found: ID does not exist" Jul 29 14:09:11.959824 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.959774 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" podStartSLOduration=0.959763406 podStartE2EDuration="959.763406ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:09:11.957954179 +0000 UTC m=+485.159455367" watchObservedRunningTime="2026-07-29 14:09:11.959763406 +0000 UTC m=+485.161264577" Jul 29 14:09:11.973156 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.973133 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk"] Jul 29 14:09:11.978486 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:11.978469 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8v7fk"] Jul 29 14:09:13.381410 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:13.381379 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9889afa9-db76-496d-aa5d-a7e82c981a0f" path="/var/lib/kubelet/pods/9889afa9-db76-496d-aa5d-a7e82c981a0f/volumes" Jul 29 14:09:22.944849 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:22.944819 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:09:23.000221 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:23.000187 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b"] Jul 29 14:09:23.000448 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:23.000423 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" podUID="952a0d09-9fae-4306-a838-7132020fb567" containerName="manager" containerID="cri-o://f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5" gracePeriod=10 Jul 29 14:09:23.897724 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:23.897691 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:09:24.017477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.017395 2560 generic.go:356] "Generic (PLEG): container finished" podID="952a0d09-9fae-4306-a838-7132020fb567" containerID="f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5" exitCode=0 Jul 29 14:09:24.017477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.017452 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" Jul 29 14:09:24.017934 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.017477 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" event={"ID":"952a0d09-9fae-4306-a838-7132020fb567","Type":"ContainerDied","Data":"f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5"} Jul 29 14:09:24.017934 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.017550 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b" event={"ID":"952a0d09-9fae-4306-a838-7132020fb567","Type":"ContainerDied","Data":"77053433ef442a66bb8f31183e3cf2d55daab520848bfd266a645717d9ab0c6a"} Jul 29 14:09:24.017934 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.017566 2560 scope.go:117] "RemoveContainer" containerID="f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5" Jul 29 14:09:24.024318 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.024295 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a0d09-9fae-4306-a838-7132020fb567-extensions-socket-volume\") pod \"952a0d09-9fae-4306-a838-7132020fb567\" (UID: \"952a0d09-9fae-4306-a838-7132020fb567\") " Jul 29 14:09:24.024428 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.024346 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dt6k\" (UniqueName: \"kubernetes.io/projected/952a0d09-9fae-4306-a838-7132020fb567-kube-api-access-6dt6k\") pod \"952a0d09-9fae-4306-a838-7132020fb567\" (UID: \"952a0d09-9fae-4306-a838-7132020fb567\") " Jul 29 14:09:24.024741 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.024716 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952a0d09-9fae-4306-a838-7132020fb567-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "952a0d09-9fae-4306-a838-7132020fb567" (UID: "952a0d09-9fae-4306-a838-7132020fb567"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:09:24.026523 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.026499 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952a0d09-9fae-4306-a838-7132020fb567-kube-api-access-6dt6k" (OuterVolumeSpecName: "kube-api-access-6dt6k") pod "952a0d09-9fae-4306-a838-7132020fb567" (UID: "952a0d09-9fae-4306-a838-7132020fb567"). InnerVolumeSpecName "kube-api-access-6dt6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:09:24.027078 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.027064 2560 scope.go:117] "RemoveContainer" containerID="f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5" Jul 29 14:09:24.027307 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:09:24.027290 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5\": container with ID starting with f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5 not found: ID does not exist" containerID="f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5" Jul 29 14:09:24.027362 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.027315 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5"} err="failed to get container status \"f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5\": rpc error: code = NotFound desc = could not find container \"f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5\": container with ID starting with f00faec8d5da014d0340509daf6a95d8ba6714f5068e1b14c5e1a785da6580d5 not found: ID does not exist" Jul 29 14:09:24.125336 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.125302 2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a0d09-9fae-4306-a838-7132020fb567-extensions-socket-volume\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:09:24.125336 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.125331 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dt6k\" (UniqueName: \"kubernetes.io/projected/952a0d09-9fae-4306-a838-7132020fb567-kube-api-access-6dt6k\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:09:24.343935 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.343903 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b"] Jul 29 14:09:24.350315 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:24.350281 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zz64b"] Jul 29 14:09:25.381016 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:25.380970 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="952a0d09-9fae-4306-a838-7132020fb567" path="/var/lib/kubelet/pods/952a0d09-9fae-4306-a838-7132020fb567/volumes" Jul 29 14:09:27.293396 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.293366 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l"] Jul 29 14:09:27.293805 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.293767 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9889afa9-db76-496d-aa5d-a7e82c981a0f" containerName="manager" Jul 29 14:09:27.293805 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.293779 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9889afa9-db76-496d-aa5d-a7e82c981a0f" containerName="manager" Jul 29 14:09:27.293805 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.293793 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952a0d09-9fae-4306-a838-7132020fb567" containerName="manager" Jul 29 14:09:27.293805 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.293799 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="952a0d09-9fae-4306-a838-7132020fb567" containerName="manager" Jul 29 14:09:27.294019 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.293870 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="952a0d09-9fae-4306-a838-7132020fb567" containerName="manager" Jul 29 14:09:27.294019 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.293880 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9889afa9-db76-496d-aa5d-a7e82c981a0f" containerName="manager" Jul 29 14:09:27.395902 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.395858 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.398636 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.398613 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 14:09:27.398952 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.398857 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n7c72\"" Jul 29 14:09:27.402994 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.402969 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l"] Jul 29 14:09:27.449832 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.449802 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.450067 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.450046 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab85095-f633-4d39-aebd-3716781e6c66-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.450183 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.450169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.450315 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.450300 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab85095-f633-4d39-aebd-3716781e6c66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.450433 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.450418 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.450568 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.450533 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.450679 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.450666 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab85095-f633-4d39-aebd-3716781e6c66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.450780 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.450767 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.450883 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.450870 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tkxf\" (UniqueName: \"kubernetes.io/projected/cab85095-f633-4d39-aebd-3716781e6c66-kube-api-access-7tkxf\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.551828 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.551747 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab85095-f633-4d39-aebd-3716781e6c66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.551828 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.551786 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.551828 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.551816 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552093 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.551833 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab85095-f633-4d39-aebd-3716781e6c66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552093 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.551997 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552093 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.552042 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tkxf\" (UniqueName: \"kubernetes.io/projected/cab85095-f633-4d39-aebd-3716781e6c66-kube-api-access-7tkxf\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552257 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.552209 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552314 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.552269 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552314 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.552281 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552424 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.552314 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab85095-f633-4d39-aebd-3716781e6c66-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552424 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.552337 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.552508 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552704 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.552684 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.552799 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.552773 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab85095-f633-4d39-aebd-3716781e6c66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.554710 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.554688 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab85095-f633-4d39-aebd-3716781e6c66-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.554945 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.554925 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab85095-f633-4d39-aebd-3716781e6c66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.560178 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.560149 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tkxf\" (UniqueName: \"kubernetes.io/projected/cab85095-f633-4d39-aebd-3716781e6c66-kube-api-access-7tkxf\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.560267 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.560223 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab85095-f633-4d39-aebd-3716781e6c66-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zv49l\" (UID: \"cab85095-f633-4d39-aebd-3716781e6c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.708148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.708114 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:27.907196 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:27.907161 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l"] Jul 29 14:09:27.908685 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:09:27.908658 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab85095_f633_4d39_aebd_3716781e6c66.slice/crio-628dc7cedb00c9985a281aae87211c1754825de85c3e9b779af973c356faa396 WatchSource:0}: Error finding container 628dc7cedb00c9985a281aae87211c1754825de85c3e9b779af973c356faa396: Status 404 returned error can't find the container with id 628dc7cedb00c9985a281aae87211c1754825de85c3e9b779af973c356faa396 Jul 29 14:09:28.043377 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:28.043345 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" event={"ID":"cab85095-f633-4d39-aebd-3716781e6c66","Type":"ContainerStarted","Data":"628dc7cedb00c9985a281aae87211c1754825de85c3e9b779af973c356faa396"} Jul 29 14:09:30.951281 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:30.951248 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:09:30.951599 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:30.951316 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:09:30.951599 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:30.951348 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:09:31.066846 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:31.066809 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" event={"ID":"cab85095-f633-4d39-aebd-3716781e6c66","Type":"ContainerStarted","Data":"83827782fb8b3f9815b185caf9995c260757bf0c23d34b33c3ea943931bdd25b"} Jul 29 14:09:31.089227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:31.089177 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" podStartSLOduration=1.048863171 podStartE2EDuration="4.089160689s" totalImagesPullingTime="3.040297518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:09:31.088404381 +0000 UTC m=+504.289905553" watchObservedRunningTime="2026-07-29 14:09:31.089160689 +0000 UTC m=+504.290661860" Jul 29 14:09:31.709146 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:31.709112 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:32.713598 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:32.713565 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:33.078256 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:33.078175 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:33.079139 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:33.079118 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:09:41.201826 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.201787 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b2xxz"] Jul 29 14:09:41.205637 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.205614 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:41.207976 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.207950 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 14:09:41.212695 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.212671 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b2xxz"] Jul 29 14:09:41.280283 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.280242 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-config-file\") pod \"limitador-limitador-64c94fd64f-b2xxz\" (UID: \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:41.280475 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.280324 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xg2\" (UniqueName: \"kubernetes.io/projected/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-kube-api-access-h2xg2\") pod \"limitador-limitador-64c94fd64f-b2xxz\" (UID: \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:41.302327 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.302282 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b2xxz"] Jul 29 14:09:41.381453 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.381424 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xg2\" (UniqueName: \"kubernetes.io/projected/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-kube-api-access-h2xg2\") pod \"limitador-limitador-64c94fd64f-b2xxz\" (UID: \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:41.381721 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.381493 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-config-file\") pod \"limitador-limitador-64c94fd64f-b2xxz\" (UID: \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:41.382162 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.382141 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-config-file\") pod \"limitador-limitador-64c94fd64f-b2xxz\" (UID: \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:41.391163 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.391135 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xg2\" (UniqueName: \"kubernetes.io/projected/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-kube-api-access-h2xg2\") pod \"limitador-limitador-64c94fd64f-b2xxz\" (UID: \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:41.517718 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.517630 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:41.656399 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.656368 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:09:41.661394 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.661214 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:41.662199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.662178 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b2xxz"] Jul 29 14:09:41.665106 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:09:41.665078 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b16feb2_8148_4c26_b5b9_259f1e8c58ee.slice/crio-3967a7baae18447592b82fdc576b98be0454354f0d37a1f5b23c44f7adda6c8e WatchSource:0}: Error finding container 3967a7baae18447592b82fdc576b98be0454354f0d37a1f5b23c44f7adda6c8e: Status 404 returned error can't find the container with id 3967a7baae18447592b82fdc576b98be0454354f0d37a1f5b23c44f7adda6c8e Jul 29 14:09:41.668453 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.668426 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:09:41.693582 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.693551 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:09:41.786223 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.786130 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/21ae4806-319b-4ddc-a6ce-f5756085e0ac-config-file\") pod \"limitador-limitador-54d6d988d5-jrpx2\" (UID: \"21ae4806-319b-4ddc-a6ce-f5756085e0ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:41.786223 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.786169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7vcd\" (UniqueName: \"kubernetes.io/projected/21ae4806-319b-4ddc-a6ce-f5756085e0ac-kube-api-access-h7vcd\") pod \"limitador-limitador-54d6d988d5-jrpx2\" (UID: \"21ae4806-319b-4ddc-a6ce-f5756085e0ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:41.886627 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.886595 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/21ae4806-319b-4ddc-a6ce-f5756085e0ac-config-file\") pod \"limitador-limitador-54d6d988d5-jrpx2\" (UID: \"21ae4806-319b-4ddc-a6ce-f5756085e0ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:41.886800 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.886637 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7vcd\" (UniqueName: \"kubernetes.io/projected/21ae4806-319b-4ddc-a6ce-f5756085e0ac-kube-api-access-h7vcd\") pod \"limitador-limitador-54d6d988d5-jrpx2\" (UID: \"21ae4806-319b-4ddc-a6ce-f5756085e0ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:41.887228 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.887211 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/21ae4806-319b-4ddc-a6ce-f5756085e0ac-config-file\") pod \"limitador-limitador-54d6d988d5-jrpx2\" (UID: \"21ae4806-319b-4ddc-a6ce-f5756085e0ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:41.894908 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.894882 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7vcd\" (UniqueName: \"kubernetes.io/projected/21ae4806-319b-4ddc-a6ce-f5756085e0ac-kube-api-access-h7vcd\") pod \"limitador-limitador-54d6d988d5-jrpx2\" (UID: \"21ae4806-319b-4ddc-a6ce-f5756085e0ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:41.982744 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:41.982713 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:42.130668 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:42.130643 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:09:42.132475 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:09:42.132447 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ae4806_319b_4ddc_a6ce_f5756085e0ac.slice/crio-4c34ae33637cf3c5f9be409b6733aa4caedd25ca806aa135b9145d88000a5ff5 WatchSource:0}: Error finding container 4c34ae33637cf3c5f9be409b6733aa4caedd25ca806aa135b9145d88000a5ff5: Status 404 returned error can't find the container with id 4c34ae33637cf3c5f9be409b6733aa4caedd25ca806aa135b9145d88000a5ff5 Jul 29 14:09:42.132612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:42.132574 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" event={"ID":"8b16feb2-8148-4c26-b5b9-259f1e8c58ee","Type":"ContainerStarted","Data":"3967a7baae18447592b82fdc576b98be0454354f0d37a1f5b23c44f7adda6c8e"} Jul 29 14:09:43.143226 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:43.143183 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" event={"ID":"21ae4806-319b-4ddc-a6ce-f5756085e0ac","Type":"ContainerStarted","Data":"4c34ae33637cf3c5f9be409b6733aa4caedd25ca806aa135b9145d88000a5ff5"} Jul 29 14:09:45.157876 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:45.157838 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" event={"ID":"8b16feb2-8148-4c26-b5b9-259f1e8c58ee","Type":"ContainerStarted","Data":"3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b"} Jul 29 14:09:45.158311 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:45.157899 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:45.159395 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:45.159369 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" event={"ID":"21ae4806-319b-4ddc-a6ce-f5756085e0ac","Type":"ContainerStarted","Data":"80997b73edfdbd8fd93e7dd4b53d950cad177bb08f98cc2cf837a776018bd835"} Jul 29 14:09:45.159523 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:45.159499 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:45.174264 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:45.174224 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" podStartSLOduration=1.263307668 podStartE2EDuration="4.174210786s" totalImagesPullingTime="2.910903118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:09:45.172847369 +0000 UTC m=+518.374348544" watchObservedRunningTime="2026-07-29 14:09:45.174210786 +0000 UTC m=+518.375711958" Jul 29 14:09:45.187363 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:45.187326 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" podStartSLOduration=1.735682835 podStartE2EDuration="4.187315184s" totalImagesPullingTime="2.451632349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:09:45.18628908 +0000 UTC m=+518.387790252" watchObservedRunningTime="2026-07-29 14:09:45.187315184 +0000 UTC m=+518.388816357" Jul 29 14:09:56.166084 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:56.166055 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:56.166582 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:56.166127 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jrpx2" Jul 29 14:09:56.213969 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:56.213932 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b2xxz"] Jul 29 14:09:56.221111 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:56.221050 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" podUID="8b16feb2-8148-4c26-b5b9-259f1e8c58ee" containerName="limitador" containerID="cri-o://3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b" gracePeriod=30 Jul 29 14:09:57.178214 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.178188 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:57.228800 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.228770 2560 generic.go:356] "Generic (PLEG): container finished" podID="8b16feb2-8148-4c26-b5b9-259f1e8c58ee" containerID="3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b" exitCode=0 Jul 29 14:09:57.228990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.228819 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" Jul 29 14:09:57.228990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.228849 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" event={"ID":"8b16feb2-8148-4c26-b5b9-259f1e8c58ee","Type":"ContainerDied","Data":"3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b"} Jul 29 14:09:57.228990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.228902 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-b2xxz" event={"ID":"8b16feb2-8148-4c26-b5b9-259f1e8c58ee","Type":"ContainerDied","Data":"3967a7baae18447592b82fdc576b98be0454354f0d37a1f5b23c44f7adda6c8e"} Jul 29 14:09:57.228990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.228927 2560 scope.go:117] "RemoveContainer" containerID="3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b" Jul 29 14:09:57.238175 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.238155 2560 scope.go:117] "RemoveContainer" containerID="3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b" Jul 29 14:09:57.238445 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:09:57.238427 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b\": container with ID starting with 3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b not found: ID does not exist" containerID="3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b" Jul 29 14:09:57.238514 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.238454 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b"} err="failed to get container status \"3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b\": rpc error: code = NotFound desc = could not find container \"3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b\": container with ID starting with 3e5c2e9b876e6b6586ee45346082b3c48f8578e3568c1fff56a03b6fbfc51f8b not found: ID does not exist" Jul 29 14:09:57.332086 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.332010 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-config-file\") pod \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\" (UID: \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\") " Jul 29 14:09:57.332228 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.332100 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2xg2\" (UniqueName: \"kubernetes.io/projected/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-kube-api-access-h2xg2\") pod \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\" (UID: \"8b16feb2-8148-4c26-b5b9-259f1e8c58ee\") " Jul 29 14:09:57.332409 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.332383 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-config-file" (OuterVolumeSpecName: "config-file") pod "8b16feb2-8148-4c26-b5b9-259f1e8c58ee" (UID: "8b16feb2-8148-4c26-b5b9-259f1e8c58ee"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:09:57.334235 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.334218 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-kube-api-access-h2xg2" (OuterVolumeSpecName: "kube-api-access-h2xg2") pod "8b16feb2-8148-4c26-b5b9-259f1e8c58ee" (UID: "8b16feb2-8148-4c26-b5b9-259f1e8c58ee"). InnerVolumeSpecName "kube-api-access-h2xg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:09:57.433680 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.433650 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2xg2\" (UniqueName: \"kubernetes.io/projected/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-kube-api-access-h2xg2\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:09:57.433680 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.433673 2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b16feb2-8148-4c26-b5b9-259f1e8c58ee-config-file\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:09:57.545181 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.545148 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b2xxz"] Jul 29 14:09:57.548506 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:57.548482 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b2xxz"] Jul 29 14:09:58.926450 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.926380 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq"] Jul 29 14:09:58.926806 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.926780 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b16feb2-8148-4c26-b5b9-259f1e8c58ee" containerName="limitador" Jul 29 14:09:58.926806 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.926792 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b16feb2-8148-4c26-b5b9-259f1e8c58ee" containerName="limitador" Jul 29 14:09:58.926877 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.926872 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b16feb2-8148-4c26-b5b9-259f1e8c58ee" containerName="limitador" Jul 29 14:09:58.931272 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.931244 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:09:58.933984 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.933963 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-s4ztv\"" Jul 29 14:09:58.934684 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.934669 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 14:09:58.934788 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.934745 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 14:09:58.934788 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.934777 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 14:09:58.944364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:58.944346 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq"] Jul 29 14:09:59.049390 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:59.049350 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3df5a58a-b2dc-4e91-9b7b-d082b01d2acc-data\") pod \"postgres-69fb4594cf-nk7rq\" (UID: \"3df5a58a-b2dc-4e91-9b7b-d082b01d2acc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:09:59.049390 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:59.049394 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt5bf\" (UniqueName: \"kubernetes.io/projected/3df5a58a-b2dc-4e91-9b7b-d082b01d2acc-kube-api-access-qt5bf\") pod \"postgres-69fb4594cf-nk7rq\" (UID: \"3df5a58a-b2dc-4e91-9b7b-d082b01d2acc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:09:59.150305 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:59.150272 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3df5a58a-b2dc-4e91-9b7b-d082b01d2acc-data\") pod \"postgres-69fb4594cf-nk7rq\" (UID: \"3df5a58a-b2dc-4e91-9b7b-d082b01d2acc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:09:59.150305 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:59.150311 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt5bf\" (UniqueName: \"kubernetes.io/projected/3df5a58a-b2dc-4e91-9b7b-d082b01d2acc-kube-api-access-qt5bf\") pod \"postgres-69fb4594cf-nk7rq\" (UID: \"3df5a58a-b2dc-4e91-9b7b-d082b01d2acc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:09:59.150716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:59.150696 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3df5a58a-b2dc-4e91-9b7b-d082b01d2acc-data\") pod \"postgres-69fb4594cf-nk7rq\" (UID: \"3df5a58a-b2dc-4e91-9b7b-d082b01d2acc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:09:59.159836 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:59.159812 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt5bf\" (UniqueName: \"kubernetes.io/projected/3df5a58a-b2dc-4e91-9b7b-d082b01d2acc-kube-api-access-qt5bf\") pod \"postgres-69fb4594cf-nk7rq\" (UID: \"3df5a58a-b2dc-4e91-9b7b-d082b01d2acc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:09:59.242084 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:59.242004 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:09:59.382155 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:59.382125 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b16feb2-8148-4c26-b5b9-259f1e8c58ee" path="/var/lib/kubelet/pods/8b16feb2-8148-4c26-b5b9-259f1e8c58ee/volumes" Jul 29 14:09:59.382477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:09:59.382459 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq"] Jul 29 14:09:59.383325 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:09:59.383301 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df5a58a_b2dc_4e91_9b7b_d082b01d2acc.slice/crio-bd3cd725f30cda26f3aab769408dd04bee7152776242615e7cfde3d5e14678b5 WatchSource:0}: Error finding container bd3cd725f30cda26f3aab769408dd04bee7152776242615e7cfde3d5e14678b5: Status 404 returned error can't find the container with id bd3cd725f30cda26f3aab769408dd04bee7152776242615e7cfde3d5e14678b5 Jul 29 14:10:00.249019 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:00.248983 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" event={"ID":"3df5a58a-b2dc-4e91-9b7b-d082b01d2acc","Type":"ContainerStarted","Data":"bd3cd725f30cda26f3aab769408dd04bee7152776242615e7cfde3d5e14678b5"} Jul 29 14:10:05.284443 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:05.284401 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" event={"ID":"3df5a58a-b2dc-4e91-9b7b-d082b01d2acc","Type":"ContainerStarted","Data":"2847bc8c89ce5be68a764b8fb6899521e6331102f2487f62effa91bb2fe705f0"} Jul 29 14:10:05.284896 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:05.284505 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:10:05.300823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:05.300767 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" podStartSLOduration=1.9601575260000001 podStartE2EDuration="7.300752263s" totalImagesPullingTime="5.340594737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:10:05.298745101 +0000 UTC m=+538.500246270" watchObservedRunningTime="2026-07-29 14:10:05.300752263 +0000 UTC m=+538.502253434" Jul 29 14:10:11.325141 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:11.325112 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nk7rq" Jul 29 14:10:12.209198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.209169 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6db4b65c74-jcxsz"] Jul 29 14:10:12.212298 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.212281 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:10:12.214806 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.214783 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7hsb5\"" Jul 29 14:10:12.215678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.215661 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 14:10:12.220702 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.220682 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6db4b65c74-jcxsz"] Jul 29 14:10:12.261937 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.261910 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4nr\" (UniqueName: \"kubernetes.io/projected/65084ef4-a1d0-4870-8f46-9e0416fb186e-kube-api-access-lg4nr\") pod \"authorino-6db4b65c74-jcxsz\" (UID: \"65084ef4-a1d0-4870-8f46-9e0416fb186e\") " pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:10:12.262091 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.261948 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65084ef4-a1d0-4870-8f46-9e0416fb186e-tls-cert\") pod \"authorino-6db4b65c74-jcxsz\" (UID: \"65084ef4-a1d0-4870-8f46-9e0416fb186e\") " pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:10:12.362880 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.362840 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4nr\" (UniqueName: \"kubernetes.io/projected/65084ef4-a1d0-4870-8f46-9e0416fb186e-kube-api-access-lg4nr\") pod \"authorino-6db4b65c74-jcxsz\" (UID: \"65084ef4-a1d0-4870-8f46-9e0416fb186e\") " pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:10:12.363278 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.362885 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65084ef4-a1d0-4870-8f46-9e0416fb186e-tls-cert\") pod \"authorino-6db4b65c74-jcxsz\" (UID: \"65084ef4-a1d0-4870-8f46-9e0416fb186e\") " pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:10:12.365459 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.365437 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65084ef4-a1d0-4870-8f46-9e0416fb186e-tls-cert\") pod \"authorino-6db4b65c74-jcxsz\" (UID: \"65084ef4-a1d0-4870-8f46-9e0416fb186e\") " pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:10:12.370679 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.370652 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4nr\" (UniqueName: \"kubernetes.io/projected/65084ef4-a1d0-4870-8f46-9e0416fb186e-kube-api-access-lg4nr\") pod \"authorino-6db4b65c74-jcxsz\" (UID: \"65084ef4-a1d0-4870-8f46-9e0416fb186e\") " pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:10:12.522819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.522742 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:10:12.658721 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:12.658690 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6db4b65c74-jcxsz"] Jul 29 14:10:12.660149 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:10:12.660124 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65084ef4_a1d0_4870_8f46_9e0416fb186e.slice/crio-d46db99226b792aae01d874dcf7d6adcbafb31855974a51db12b75e655cceb10 WatchSource:0}: Error finding container d46db99226b792aae01d874dcf7d6adcbafb31855974a51db12b75e655cceb10: Status 404 returned error can't find the container with id d46db99226b792aae01d874dcf7d6adcbafb31855974a51db12b75e655cceb10 Jul 29 14:10:13.340617 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:13.340557 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" event={"ID":"65084ef4-a1d0-4870-8f46-9e0416fb186e","Type":"ContainerStarted","Data":"d46db99226b792aae01d874dcf7d6adcbafb31855974a51db12b75e655cceb10"} Jul 29 14:10:16.360183 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:16.360149 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" event={"ID":"65084ef4-a1d0-4870-8f46-9e0416fb186e","Type":"ContainerStarted","Data":"2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426"} Jul 29 14:10:16.380601 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:16.380554 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" podStartSLOduration=1.374814073 podStartE2EDuration="4.380522839s" totalImagesPullingTime="3.005708766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:10:16.378590399 +0000 UTC m=+549.580091571" watchObservedRunningTime="2026-07-29 14:10:16.380522839 +0000 UTC m=+549.582024010" Jul 29 14:10:57.305783 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:10:57.305738 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:11:04.120735 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:04.120686 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zv49l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 14:11:04.121159 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:04.120784 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" podUID="cab85095-f633-4d39-aebd-3716781e6c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 14:11:05.649555 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:05.649512 2560 generic.go:356] "Generic (PLEG): container finished" podID="cab85095-f633-4d39-aebd-3716781e6c66" containerID="83827782fb8b3f9815b185caf9995c260757bf0c23d34b33c3ea943931bdd25b" exitCode=0 Jul 29 14:11:05.649981 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:05.649592 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" event={"ID":"cab85095-f633-4d39-aebd-3716781e6c66","Type":"ContainerDied","Data":"83827782fb8b3f9815b185caf9995c260757bf0c23d34b33c3ea943931bdd25b"} Jul 29 14:11:05.650029 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:05.649982 2560 scope.go:117] "RemoveContainer" containerID="83827782fb8b3f9815b185caf9995c260757bf0c23d34b33c3ea943931bdd25b" Jul 29 14:11:05.651094 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:05.651064 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:11:05.651199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:05.651148 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:11:05.651199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:05.651193 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:11:06.657477 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:06.657439 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" event={"ID":"cab85095-f633-4d39-aebd-3716781e6c66","Type":"ContainerStarted","Data":"283ca420a81f9d0394e046851d3bea48397b71f3a8c53e27c214bdb0008475ca"} Jul 29 14:11:06.709084 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:06.709047 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:11:06.709288 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:06.709263 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zv49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 29 14:11:06.709334 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:06.709316 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" podUID="cab85095-f633-4d39-aebd-3716781e6c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 29 14:11:07.346783 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:07.346756 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:11:07.350036 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:07.350012 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:11:07.350167 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:07.350110 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:11:07.353654 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:07.353634 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:11:07.708994 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:07.708943 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zv49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 29 14:11:07.709518 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:07.709031 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" podUID="cab85095-f633-4d39-aebd-3716781e6c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 29 14:11:08.709134 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:08.709093 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zv49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 29 14:11:08.710243 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:08.709830 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" podUID="cab85095-f633-4d39-aebd-3716781e6c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 29 14:11:09.709528 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:09.709487 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zv49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 29 14:11:09.710141 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:09.710105 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" podUID="cab85095-f633-4d39-aebd-3716781e6c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 29 14:11:10.709272 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:10.708826 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zv49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 29 14:11:10.709272 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:10.708888 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" podUID="cab85095-f633-4d39-aebd-3716781e6c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 29 14:11:11.711685 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:11.711657 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:11:11.712099 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:11.711940 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:11:11.712685 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:11.712666 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zv49l" Jul 29 14:11:16.647428 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:16.647389 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:11:25.143507 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.143473 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw"] Jul 29 14:11:25.147057 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.147035 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.149343 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.149299 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 14:11:25.150429 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.150408 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 29 14:11:25.150429 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.150421 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 14:11:25.150615 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.150430 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nccq4\"" Jul 29 14:11:25.162050 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.162027 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw"] Jul 29 14:11:25.261315 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.261282 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jst8b\" (UniqueName: \"kubernetes.io/projected/a95663aa-9693-4fd4-9621-1222214c08d1-kube-api-access-jst8b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.261315 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.261317 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.261521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.261353 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a95663aa-9693-4fd4-9621-1222214c08d1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.261521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.261418 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.261521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.261461 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.261521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.261493 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.362025 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.361988 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.362189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.362048 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jst8b\" (UniqueName: \"kubernetes.io/projected/a95663aa-9693-4fd4-9621-1222214c08d1-kube-api-access-jst8b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.362189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.362068 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.362189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.362091 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a95663aa-9693-4fd4-9621-1222214c08d1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.362189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.362126 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.362370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.362345 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.362519 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.362497 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.362639 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.362564 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.362756 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.362735 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.364431 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.364410 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a95663aa-9693-4fd4-9621-1222214c08d1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.364694 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.364677 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a95663aa-9693-4fd4-9621-1222214c08d1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.369666 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.369650 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jst8b\" (UniqueName: \"kubernetes.io/projected/a95663aa-9693-4fd4-9621-1222214c08d1-kube-api-access-jst8b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-l7fqw\" (UID: \"a95663aa-9693-4fd4-9621-1222214c08d1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.457781 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.457749 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:25.603115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.603084 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw"] Jul 29 14:11:25.604005 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:11:25.603980 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95663aa_9693_4fd4_9621_1222214c08d1.slice/crio-e4d7682f846eb72110f30f3cde14b46fab384b4b75ddf7b5295a85b6980bb45b WatchSource:0}: Error finding container e4d7682f846eb72110f30f3cde14b46fab384b4b75ddf7b5295a85b6980bb45b: Status 404 returned error can't find the container with id e4d7682f846eb72110f30f3cde14b46fab384b4b75ddf7b5295a85b6980bb45b Jul 29 14:11:25.605909 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.605890 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:11:25.764071 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.763983 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" event={"ID":"a95663aa-9693-4fd4-9621-1222214c08d1","Type":"ContainerStarted","Data":"e4d7682f846eb72110f30f3cde14b46fab384b4b75ddf7b5295a85b6980bb45b"} Jul 29 14:11:25.947122 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:25.947091 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:11:27.784006 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:27.783961 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" event={"ID":"a95663aa-9693-4fd4-9621-1222214c08d1","Type":"ContainerStarted","Data":"e5bf72ea4cd2080e06f80b2e4b9be9a202c284481de8280ea82fb13885aedea1"} Jul 29 14:11:27.805833 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:27.805786 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" podStartSLOduration=1.585417777 podStartE2EDuration="2.805771693s" totalImagesPullingTime="1.220353916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:11:27.803458704 +0000 UTC m=+621.004959877" watchObservedRunningTime="2026-07-29 14:11:27.805771693 +0000 UTC m=+621.007272843" Jul 29 14:11:35.458051 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:35.458010 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:35.458518 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:35.458074 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:35.470788 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:35.470765 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:35.842796 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:35.842711 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-l7fqw" Jul 29 14:11:36.642481 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:36.642445 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:11:53.453745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:11:53.453704 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:12:31.744332 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:12:31.744295 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:12:47.252237 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:12:47.252202 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:13:11.372646 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.372614 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-758465fbff-nb9c6"] Jul 29 14:13:11.376388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.376362 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758465fbff-nb9c6" Jul 29 14:13:11.383920 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.383894 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-758465fbff-nb9c6"] Jul 29 14:13:11.556839 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.556806 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29c73cb9-ff52-4c18-9aca-948a80c74419-tls-cert\") pod \"authorino-758465fbff-nb9c6\" (UID: \"29c73cb9-ff52-4c18-9aca-948a80c74419\") " pod="kuadrant-system/authorino-758465fbff-nb9c6" Jul 29 14:13:11.557003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.556847 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkzvr\" (UniqueName: \"kubernetes.io/projected/29c73cb9-ff52-4c18-9aca-948a80c74419-kube-api-access-dkzvr\") pod \"authorino-758465fbff-nb9c6\" (UID: \"29c73cb9-ff52-4c18-9aca-948a80c74419\") " pod="kuadrant-system/authorino-758465fbff-nb9c6" Jul 29 14:13:11.657513 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.657430 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29c73cb9-ff52-4c18-9aca-948a80c74419-tls-cert\") pod \"authorino-758465fbff-nb9c6\" (UID: \"29c73cb9-ff52-4c18-9aca-948a80c74419\") " pod="kuadrant-system/authorino-758465fbff-nb9c6" Jul 29 14:13:11.657513 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.657465 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzvr\" (UniqueName: \"kubernetes.io/projected/29c73cb9-ff52-4c18-9aca-948a80c74419-kube-api-access-dkzvr\") pod \"authorino-758465fbff-nb9c6\" (UID: \"29c73cb9-ff52-4c18-9aca-948a80c74419\") " pod="kuadrant-system/authorino-758465fbff-nb9c6" Jul 29 14:13:11.660072 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.660048 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29c73cb9-ff52-4c18-9aca-948a80c74419-tls-cert\") pod \"authorino-758465fbff-nb9c6\" (UID: \"29c73cb9-ff52-4c18-9aca-948a80c74419\") " pod="kuadrant-system/authorino-758465fbff-nb9c6" Jul 29 14:13:11.666077 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.666054 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzvr\" (UniqueName: \"kubernetes.io/projected/29c73cb9-ff52-4c18-9aca-948a80c74419-kube-api-access-dkzvr\") pod \"authorino-758465fbff-nb9c6\" (UID: \"29c73cb9-ff52-4c18-9aca-948a80c74419\") " pod="kuadrant-system/authorino-758465fbff-nb9c6" Jul 29 14:13:11.688192 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.688162 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758465fbff-nb9c6" Jul 29 14:13:11.829068 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:11.829047 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-758465fbff-nb9c6"] Jul 29 14:13:11.830750 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:13:11.830722 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29c73cb9_ff52_4c18_9aca_948a80c74419.slice/crio-741d360466ad62000aa70239bf99b60e0a9fd2259b65ac88c899439bdcf450fa WatchSource:0}: Error finding container 741d360466ad62000aa70239bf99b60e0a9fd2259b65ac88c899439bdcf450fa: Status 404 returned error can't find the container with id 741d360466ad62000aa70239bf99b60e0a9fd2259b65ac88c899439bdcf450fa Jul 29 14:13:12.387340 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:12.387309 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758465fbff-nb9c6" event={"ID":"29c73cb9-ff52-4c18-9aca-948a80c74419","Type":"ContainerStarted","Data":"741d360466ad62000aa70239bf99b60e0a9fd2259b65ac88c899439bdcf450fa"} Jul 29 14:13:13.398121 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.398071 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758465fbff-nb9c6" event={"ID":"29c73cb9-ff52-4c18-9aca-948a80c74419","Type":"ContainerStarted","Data":"bd47668ccbac3b096e71ddfad20b3b49555f841662d5542c9269f433f830bf1f"} Jul 29 14:13:13.417806 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.417750 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-758465fbff-nb9c6" podStartSLOduration=1.349103067 podStartE2EDuration="2.417731782s" totalImagesPullingTime="1.068628715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:13:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:13:13.416270355 +0000 UTC m=+726.617771553" watchObservedRunningTime="2026-07-29 14:13:13.417731782 +0000 UTC m=+726.619232953" Jul 29 14:13:13.447975 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.447930 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6db4b65c74-jcxsz"] Jul 29 14:13:13.448218 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.448162 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" podUID="65084ef4-a1d0-4870-8f46-9e0416fb186e" containerName="authorino" containerID="cri-o://2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426" gracePeriod=30 Jul 29 14:13:13.722971 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.722947 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:13:13.878729 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.878688 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg4nr\" (UniqueName: \"kubernetes.io/projected/65084ef4-a1d0-4870-8f46-9e0416fb186e-kube-api-access-lg4nr\") pod \"65084ef4-a1d0-4870-8f46-9e0416fb186e\" (UID: \"65084ef4-a1d0-4870-8f46-9e0416fb186e\") " Jul 29 14:13:13.878937 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.878785 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65084ef4-a1d0-4870-8f46-9e0416fb186e-tls-cert\") pod \"65084ef4-a1d0-4870-8f46-9e0416fb186e\" (UID: \"65084ef4-a1d0-4870-8f46-9e0416fb186e\") " Jul 29 14:13:13.881519 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.881490 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65084ef4-a1d0-4870-8f46-9e0416fb186e-kube-api-access-lg4nr" (OuterVolumeSpecName: "kube-api-access-lg4nr") pod "65084ef4-a1d0-4870-8f46-9e0416fb186e" (UID: "65084ef4-a1d0-4870-8f46-9e0416fb186e"). InnerVolumeSpecName "kube-api-access-lg4nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:13:13.892955 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.892922 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65084ef4-a1d0-4870-8f46-9e0416fb186e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "65084ef4-a1d0-4870-8f46-9e0416fb186e" (UID: "65084ef4-a1d0-4870-8f46-9e0416fb186e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:13:13.980129 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.980048 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4nr\" (UniqueName: \"kubernetes.io/projected/65084ef4-a1d0-4870-8f46-9e0416fb186e-kube-api-access-lg4nr\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:13:13.980129 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:13.980076 2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65084ef4-a1d0-4870-8f46-9e0416fb186e-tls-cert\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:13:14.405161 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:14.405084 2560 generic.go:356] "Generic (PLEG): container finished" podID="65084ef4-a1d0-4870-8f46-9e0416fb186e" containerID="2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426" exitCode=0 Jul 29 14:13:14.405161 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:14.405127 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" Jul 29 14:13:14.405161 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:14.405159 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" event={"ID":"65084ef4-a1d0-4870-8f46-9e0416fb186e","Type":"ContainerDied","Data":"2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426"} Jul 29 14:13:14.405714 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:14.405190 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6db4b65c74-jcxsz" event={"ID":"65084ef4-a1d0-4870-8f46-9e0416fb186e","Type":"ContainerDied","Data":"d46db99226b792aae01d874dcf7d6adcbafb31855974a51db12b75e655cceb10"} Jul 29 14:13:14.405714 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:14.405209 2560 scope.go:117] "RemoveContainer" containerID="2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426" Jul 29 14:13:14.417172 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:14.417155 2560 scope.go:117] "RemoveContainer" containerID="2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426" Jul 29 14:13:14.417454 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:13:14.417433 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426\": container with ID starting with 2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426 not found: ID does not exist" containerID="2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426" Jul 29 14:13:14.417512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:14.417464 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426"} err="failed to get container status \"2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426\": rpc error: code = NotFound desc = could not find container \"2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426\": container with ID starting with 2646da5687c851f51aebeff43f253171b04f027aae344eef2caf6e387b957426 not found: ID does not exist" Jul 29 14:13:14.426926 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:14.426902 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6db4b65c74-jcxsz"] Jul 29 14:13:14.434504 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:14.434474 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6db4b65c74-jcxsz"] Jul 29 14:13:15.381285 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:15.381252 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65084ef4-a1d0-4870-8f46-9e0416fb186e" path="/var/lib/kubelet/pods/65084ef4-a1d0-4870-8f46-9e0416fb186e/volumes" Jul 29 14:13:47.659854 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:47.659771 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:13:57.643609 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:13:57.643576 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:14:07.960875 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:14:07.960843 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:15:00.164355 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:00.164317 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl"] Jul 29 14:15:00.164815 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:00.164801 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65084ef4-a1d0-4870-8f46-9e0416fb186e" containerName="authorino" Jul 29 14:15:00.164863 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:00.164817 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="65084ef4-a1d0-4870-8f46-9e0416fb186e" containerName="authorino" Jul 29 14:15:00.164902 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:00.164886 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="65084ef4-a1d0-4870-8f46-9e0416fb186e" containerName="authorino" Jul 29 14:15:00.168138 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:00.168122 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl" Jul 29 14:15:00.171094 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:00.171075 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:15:00.177043 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:00.177018 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl"] Jul 29 14:15:00.180127 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:00.180106 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl" Jul 29 14:15:00.323874 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:00.323849 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl"] Jul 29 14:15:00.326437 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:15:00.326407 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdbc566_063c_4909_b252_849d94e32bb1.slice/crio-7e72c5e95120c4d20aece96ae3ef86452f8f62648dc1f7f9440b7366a223a487 WatchSource:0}: Error finding container 7e72c5e95120c4d20aece96ae3ef86452f8f62648dc1f7f9440b7366a223a487: Status 404 returned error can't find the container with id 7e72c5e95120c4d20aece96ae3ef86452f8f62648dc1f7f9440b7366a223a487 Jul 29 14:15:01.048047 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:01.048014 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl" event={"ID":"bbdbc566-063c-4909-b252-849d94e32bb1","Type":"ContainerStarted","Data":"7e72c5e95120c4d20aece96ae3ef86452f8f62648dc1f7f9440b7366a223a487"} Jul 29 14:15:04.070666 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:04.070634 2560 generic.go:356] "Generic (PLEG): container finished" podID="bbdbc566-063c-4909-b252-849d94e32bb1" containerID="51a9d98cccad90f5cd03271e5632005bb3bf423a94b150abbb61d0890b99b9b9" exitCode=0 Jul 29 14:15:04.071038 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:04.070705 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl" event={"ID":"bbdbc566-063c-4909-b252-849d94e32bb1","Type":"ContainerDied","Data":"51a9d98cccad90f5cd03271e5632005bb3bf423a94b150abbb61d0890b99b9b9"} Jul 29 14:15:05.222531 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:05.222502 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl" Jul 29 14:15:06.083597 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:06.083567 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl" Jul 29 14:15:06.083800 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:06.083600 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755575-c52sl" event={"ID":"bbdbc566-063c-4909-b252-849d94e32bb1","Type":"ContainerDied","Data":"7e72c5e95120c4d20aece96ae3ef86452f8f62648dc1f7f9440b7366a223a487"} Jul 29 14:15:06.083800 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:06.083652 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e72c5e95120c4d20aece96ae3ef86452f8f62648dc1f7f9440b7366a223a487" Jul 29 14:15:51.241743 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:15:51.241706 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:16:00.556006 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:00.555926 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:16:07.389941 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:07.389912 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:16:07.393961 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:07.393937 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:16:07.398360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:07.398340 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:16:07.401765 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:07.401744 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:16:14.451405 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:14.451368 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:16:31.867275 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:31.867237 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:16:41.551616 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:41.551580 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:16:49.463722 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:49.463682 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:16:58.042871 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:16:58.042832 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:17:06.950630 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:17:06.950590 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:17:16.653396 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:17:16.653357 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:17:27.849861 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:17:27.849828 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:17:37.550993 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:17:37.550964 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:18:00.156653 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:18:00.156613 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:18:12.548916 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:18:12.548877 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:18:35.452038 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:18:35.452004 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:18:46.154200 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:18:46.154161 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:18:56.628123 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:18:56.628087 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:19:13.992443 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:19:13.992407 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:19:23.256478 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:19:23.256441 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:19:33.288516 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:19:33.288475 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:19:43.855063 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:19:43.855027 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:19:53.351575 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:19:53.351528 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:20:18.608911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:18.608871 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn"] Jul 29 14:20:18.609325 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:18.609096 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" podUID="cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f" containerName="manager" containerID="cri-o://b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464" gracePeriod=10 Jul 29 14:20:21.571339 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.571311 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:20:21.633302 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.633271 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn45q\" (UniqueName: \"kubernetes.io/projected/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-kube-api-access-vn45q\") pod \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\" (UID: \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\") " Jul 29 14:20:21.633469 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.633411 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-extensions-socket-volume\") pod \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\" (UID: \"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f\") " Jul 29 14:20:21.633779 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.633755 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f" (UID: "cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:20:21.635734 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.635702 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-kube-api-access-vn45q" (OuterVolumeSpecName: "kube-api-access-vn45q") pod "cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f" (UID: "cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f"). InnerVolumeSpecName "kube-api-access-vn45q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:20:21.734577 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.734527 2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-extensions-socket-volume\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:20:21.734577 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.734574 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn45q\" (UniqueName: \"kubernetes.io/projected/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f-kube-api-access-vn45q\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:20:21.961327 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.961296 2560 generic.go:356] "Generic (PLEG): container finished" podID="cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f" containerID="b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464" exitCode=0 Jul 29 14:20:21.961507 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.961343 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" Jul 29 14:20:21.961507 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.961377 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" event={"ID":"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f","Type":"ContainerDied","Data":"b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464"} Jul 29 14:20:21.961507 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.961427 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn" event={"ID":"cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f","Type":"ContainerDied","Data":"1e9d7f6f74f4be030a12c4ed1e61ddf3474cf9450d2b3550b3be0a0314144e31"} Jul 29 14:20:21.961507 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.961449 2560 scope.go:117] "RemoveContainer" containerID="b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464" Jul 29 14:20:21.972332 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.972316 2560 scope.go:117] "RemoveContainer" containerID="b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464" Jul 29 14:20:21.972674 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:20:21.972651 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464\": container with ID starting with b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464 not found: ID does not exist" containerID="b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464" Jul 29 14:20:21.972750 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.972685 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464"} err="failed to get container status \"b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464\": rpc error: code = NotFound desc = could not find container \"b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464\": container with ID starting with b1811797ade7bcc1f6533962c145d3094baaff8e6b19df07b8ae5bb9fd891464 not found: ID does not exist" Jul 29 14:20:21.983696 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.983667 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn"] Jul 29 14:20:21.986565 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:21.986532 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l6fwn"] Jul 29 14:20:23.381270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:20:23.381230 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f" path="/var/lib/kubelet/pods/cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f/volumes" Jul 29 14:21:07.430029 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:07.430005 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:21:07.433509 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:07.433484 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:21:07.439753 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:07.439723 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:21:07.442935 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:07.442918 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:21:19.838731 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.838661 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl"] Jul 29 14:21:19.839294 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.839275 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f" containerName="manager" Jul 29 14:21:19.839294 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.839295 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f" containerName="manager" Jul 29 14:21:19.839411 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.839305 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbdbc566-063c-4909-b252-849d94e32bb1" containerName="cleanup" Jul 29 14:21:19.839411 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.839311 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbdbc566-063c-4909-b252-849d94e32bb1" containerName="cleanup" Jul 29 14:21:19.839411 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.839397 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9fe48f-8e6f-4db3-bfdd-67d9c20ff07f" containerName="manager" Jul 29 14:21:19.839587 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.839412 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbdbc566-063c-4909-b252-849d94e32bb1" containerName="cleanup" Jul 29 14:21:19.843729 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.843711 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:19.846318 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.846299 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2nwr\"" Jul 29 14:21:19.854635 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.854612 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl"] Jul 29 14:21:19.980320 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.980279 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjjds\" (UniqueName: \"kubernetes.io/projected/8f255ca0-8fcf-4b0d-a789-77bd878631f0-kube-api-access-xjjds\") pod \"kuadrant-operator-controller-manager-86878c769f-hxrkl\" (UID: \"8f255ca0-8fcf-4b0d-a789-77bd878631f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:19.980505 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:19.980369 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f255ca0-8fcf-4b0d-a789-77bd878631f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hxrkl\" (UID: \"8f255ca0-8fcf-4b0d-a789-77bd878631f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:20.080865 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:20.080824 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f255ca0-8fcf-4b0d-a789-77bd878631f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hxrkl\" (UID: \"8f255ca0-8fcf-4b0d-a789-77bd878631f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:20.081050 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:20.080928 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjds\" (UniqueName: \"kubernetes.io/projected/8f255ca0-8fcf-4b0d-a789-77bd878631f0-kube-api-access-xjjds\") pod \"kuadrant-operator-controller-manager-86878c769f-hxrkl\" (UID: \"8f255ca0-8fcf-4b0d-a789-77bd878631f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:20.081237 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:20.081218 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f255ca0-8fcf-4b0d-a789-77bd878631f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hxrkl\" (UID: \"8f255ca0-8fcf-4b0d-a789-77bd878631f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:20.093233 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:20.093158 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjds\" (UniqueName: \"kubernetes.io/projected/8f255ca0-8fcf-4b0d-a789-77bd878631f0-kube-api-access-xjjds\") pod \"kuadrant-operator-controller-manager-86878c769f-hxrkl\" (UID: \"8f255ca0-8fcf-4b0d-a789-77bd878631f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:20.154636 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:20.154602 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:20.310201 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:20.310171 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl"] Jul 29 14:21:20.311043 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:21:20.311019 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f255ca0_8fcf_4b0d_a789_77bd878631f0.slice/crio-f2a6b3dca324c95e3bffd25ff6488a76e84c4b3ded53fece9a166d874a6102c5 WatchSource:0}: Error finding container f2a6b3dca324c95e3bffd25ff6488a76e84c4b3ded53fece9a166d874a6102c5: Status 404 returned error can't find the container with id f2a6b3dca324c95e3bffd25ff6488a76e84c4b3ded53fece9a166d874a6102c5 Jul 29 14:21:21.303238 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:21.303202 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" event={"ID":"8f255ca0-8fcf-4b0d-a789-77bd878631f0","Type":"ContainerStarted","Data":"6d1c55383776b4a3ffd91b3af75e6d41ec66660d9492261d07c2651e5fe3084c"} Jul 29 14:21:21.303730 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:21.303259 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" event={"ID":"8f255ca0-8fcf-4b0d-a789-77bd878631f0","Type":"ContainerStarted","Data":"f2a6b3dca324c95e3bffd25ff6488a76e84c4b3ded53fece9a166d874a6102c5"} Jul 29 14:21:21.303730 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:21.303407 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:21.329209 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:21.329161 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" podStartSLOduration=2.32914673 podStartE2EDuration="2.32914673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:21:21.3276677 +0000 UTC m=+1214.529168882" watchObservedRunningTime="2026-07-29 14:21:21.32914673 +0000 UTC m=+1214.530647902" Jul 29 14:21:32.311486 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:32.311451 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxrkl" Jul 29 14:21:40.948820 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:40.948781 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:21:52.159029 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:21:52.158994 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:22:06.551858 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:22:06.551817 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:22:22.452416 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:22:22.452367 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:22:35.741706 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:22:35.741671 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:22:43.647697 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:22:43.647660 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:22:53.638564 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:22:53.638503 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:23:06.154258 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:23:06.154224 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:23:14.145382 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:23:14.145342 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:23:25.259633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:23:25.259594 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:23:36.151472 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:23:36.151394 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:23:43.457499 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:23:43.457465 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:24:01.956448 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:24:01.956408 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:24:11.551866 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:24:11.551825 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:24:31.147372 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:24:31.147337 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:25:01.352048 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:25:01.351966 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:25:16.053205 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:25:16.053164 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:25:26.044924 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:25:26.044886 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:25:43.357998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:25:43.357960 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:25:55.152527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:25:55.152487 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:26:05.153821 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:26:05.153787 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:26:07.469087 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:26:07.468981 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:26:07.473108 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:26:07.472731 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:26:07.482488 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:26:07.482467 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:26:07.486012 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:26:07.485994 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:26:13.150421 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:26:13.150380 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:26:24.153020 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:26:24.152980 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:26:35.758920 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:26:35.758832 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:26:54.765012 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:26:54.759823 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:27:12.545226 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:27:12.545191 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:27:21.054253 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:27:21.054218 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:27:33.153750 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:27:33.153709 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:27:50.255807 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:27:50.255767 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:28:02.855918 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:02.855816 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:28:15.052629 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:15.052586 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:28:22.651706 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:22.651666 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:28:37.007284 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:37.007235 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:28:46.848797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:46.848752 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:28:56.455526 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.455494 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48"] Jul 29 14:28:56.459293 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.459275 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.462217 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.462193 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d8b7adbd-openshift-default-dockercfg-wwrqd\"" Jul 29 14:28:56.486882 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.486848 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48"] Jul 29 14:28:56.555244 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.555194 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-token\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.555517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.555261 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-credential-socket\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.555517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.555288 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gzk\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-kube-api-access-r7gzk\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.555517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.555312 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-certs\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.555517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.555349 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-socket\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.555517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.555377 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-envoy\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.555517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.555396 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istiod-ca-cert\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.555517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.555417 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-podinfo\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.555517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.555445 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-data\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.656414 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656370 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-credential-socket\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.656414 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656411 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gzk\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-kube-api-access-r7gzk\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.656713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656432 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-certs\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.656713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656459 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-socket\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.656713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656514 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-envoy\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.656713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656587 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istiod-ca-cert\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.656713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-podinfo\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.656713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-data\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.657035 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656751 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-token\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.657035 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.656887 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-credential-socket\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.657035 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.657015 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-certs\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.657185 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.657092 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-data\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.657185 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.657164 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-socket\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.657286 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.657268 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istiod-ca-cert\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.659101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.659077 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-envoy\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.659284 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.659250 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-podinfo\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.665421 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.665399 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-token\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.665717 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.665695 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gzk\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-kube-api-access-r7gzk\") pod \"e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.770298 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.770201 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:56.925910 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.925878 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48"] Jul 29 14:28:56.926997 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:28:56.926972 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd815cf_af4d_4a14_ac96_f162a7011d0d.slice/crio-8b36fc8786ca04c01817b75d39d9dd077911c52e248b3b149d1d6df3f76e8141 WatchSource:0}: Error finding container 8b36fc8786ca04c01817b75d39d9dd077911c52e248b3b149d1d6df3f76e8141: Status 404 returned error can't find the container with id 8b36fc8786ca04c01817b75d39d9dd077911c52e248b3b149d1d6df3f76e8141 Jul 29 14:28:56.929219 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.929191 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:28:56.929278 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.929259 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:28:56.929315 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:56.929291 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:28:57.026079 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:57.025977 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" event={"ID":"5dd815cf-af4d-4a14-ac96-f162a7011d0d","Type":"ContainerStarted","Data":"22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7"} Jul 29 14:28:57.026079 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:57.026043 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" event={"ID":"5dd815cf-af4d-4a14-ac96-f162a7011d0d","Type":"ContainerStarted","Data":"8b36fc8786ca04c01817b75d39d9dd077911c52e248b3b149d1d6df3f76e8141"} Jul 29 14:28:57.047194 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:57.047129 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" podStartSLOduration=1.047112129 podStartE2EDuration="1.047112129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:28:57.045484387 +0000 UTC m=+1670.246985560" watchObservedRunningTime="2026-07-29 14:28:57.047112129 +0000 UTC m=+1670.248613301" Jul 29 14:28:57.155497 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:57.155460 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:28:57.771157 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:57.771118 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:57.776030 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:57.776003 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:58.032751 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:58.032653 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:28:58.033838 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:28:58.033821 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:29:10.454740 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:10.454693 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:29:11.111643 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:11.111610 2560 generic.go:356] "Generic (PLEG): container finished" podID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerID="22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7" exitCode=0 Jul 29 14:29:11.111883 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:11.111665 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" event={"ID":"5dd815cf-af4d-4a14-ac96-f162a7011d0d","Type":"ContainerDied","Data":"22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7"} Jul 29 14:29:11.112113 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:11.112099 2560 scope.go:117] "RemoveContainer" containerID="22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7" Jul 29 14:29:11.113212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:11.113178 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:29:11.113302 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:11.113283 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:29:11.113385 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:11.113335 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:29:12.121489 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:12.121449 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" event={"ID":"5dd815cf-af4d-4a14-ac96-f162a7011d0d","Type":"ContainerStarted","Data":"e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09"} Jul 29 14:29:12.770916 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:12.770872 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:29:12.772226 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:12.772202 2560 patch_prober.go:28] interesting pod/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 29 14:29:12.772296 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:12.772253 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 29 14:29:13.033533 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:13.033420 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:29:13.770843 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:13.770801 2560 patch_prober.go:28] interesting pod/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 29 14:29:13.771256 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:13.770861 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 29 14:29:14.771488 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:14.771449 2560 patch_prober.go:28] interesting pod/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 29 14:29:14.771934 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:14.771508 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 29 14:29:15.771433 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:15.771394 2560 patch_prober.go:28] interesting pod/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 29 14:29:15.771654 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:15.771456 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 29 14:29:16.775185 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:16.775152 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:29:16.776277 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:16.776257 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:29:23.950512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:23.950468 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r"] Jul 29 14:29:23.953433 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:23.953409 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" Jul 29 14:29:23.955693 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:23.955670 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:29:23.962601 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:23.962572 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r"] Jul 29 14:29:24.025492 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:24.025449 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f6120-8569-4653-8662-c468259be16d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r\" (UID: \"0f2f6120-8569-4653-8662-c468259be16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" Jul 29 14:29:24.127174 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:24.127135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f6120-8569-4653-8662-c468259be16d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r\" (UID: \"0f2f6120-8569-4653-8662-c468259be16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" Jul 29 14:29:24.127841 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:24.127822 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f6120-8569-4653-8662-c468259be16d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r\" (UID: \"0f2f6120-8569-4653-8662-c468259be16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" Jul 29 14:29:24.264916 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:24.264822 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" Jul 29 14:29:24.615338 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:24.615310 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r"] Jul 29 14:29:24.617790 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:29:24.617751 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2f6120_8569_4653_8662_c468259be16d.slice/crio-4f2ab5dab910ec169e6fc242d83ece31e6b1fbb1fbe29991db2728eb6247281a WatchSource:0}: Error finding container 4f2ab5dab910ec169e6fc242d83ece31e6b1fbb1fbe29991db2728eb6247281a: Status 404 returned error can't find the container with id 4f2ab5dab910ec169e6fc242d83ece31e6b1fbb1fbe29991db2728eb6247281a Jul 29 14:29:25.203093 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:25.203061 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" event={"ID":"0f2f6120-8569-4653-8662-c468259be16d","Type":"ContainerStarted","Data":"1c6c2ad3ed96b9b898084a176e15d97363bcf8313f9ad7f1951c00eacb88bb32"} Jul 29 14:29:25.203500 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:25.203113 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" event={"ID":"0f2f6120-8569-4653-8662-c468259be16d","Type":"ContainerStarted","Data":"4f2ab5dab910ec169e6fc242d83ece31e6b1fbb1fbe29991db2728eb6247281a"} Jul 29 14:29:25.221171 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:25.221117 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" podStartSLOduration=2.221100418 podStartE2EDuration="2.221100418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:29:25.21861023 +0000 UTC m=+1698.420111404" watchObservedRunningTime="2026-07-29 14:29:25.221100418 +0000 UTC m=+1698.422601589" Jul 29 14:29:26.210622 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:26.210575 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f2f6120-8569-4653-8662-c468259be16d" containerID="1c6c2ad3ed96b9b898084a176e15d97363bcf8313f9ad7f1951c00eacb88bb32" exitCode=0 Jul 29 14:29:26.211059 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:26.210664 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" event={"ID":"0f2f6120-8569-4653-8662-c468259be16d","Type":"ContainerDied","Data":"1c6c2ad3ed96b9b898084a176e15d97363bcf8313f9ad7f1951c00eacb88bb32"} Jul 29 14:29:27.359296 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:27.359270 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" Jul 29 14:29:27.458760 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:27.458723 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f6120-8569-4653-8662-c468259be16d-maas-api-service-ca\") pod \"0f2f6120-8569-4653-8662-c468259be16d\" (UID: \"0f2f6120-8569-4653-8662-c468259be16d\") " Jul 29 14:29:27.459133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:27.459107 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2f6120-8569-4653-8662-c468259be16d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f2f6120-8569-4653-8662-c468259be16d" (UID: "0f2f6120-8569-4653-8662-c468259be16d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:29:27.560326 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:27.560227 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f6120-8569-4653-8662-c468259be16d-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:28.223452 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:28.223370 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" Jul 29 14:29:28.223452 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:28.223406 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r" event={"ID":"0f2f6120-8569-4653-8662-c468259be16d","Type":"ContainerDied","Data":"4f2ab5dab910ec169e6fc242d83ece31e6b1fbb1fbe29991db2728eb6247281a"} Jul 29 14:29:28.223692 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:28.223460 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f2ab5dab910ec169e6fc242d83ece31e6b1fbb1fbe29991db2728eb6247281a" Jul 29 14:29:29.382356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:29.382320 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r"] Jul 29 14:29:29.382356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:29.382360 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8b7adbd-x642r"] Jul 29 14:29:29.516078 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:29.516033 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48"] Jul 29 14:29:29.516292 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:29.516265 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" containerID="cri-o://e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09" gracePeriod=30 Jul 29 14:29:31.381478 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:31.381439 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f2f6120-8569-4653-8662-c468259be16d" path="/var/lib/kubelet/pods/0f2f6120-8569-4653-8662-c468259be16d/volumes" Jul 29 14:29:31.777704 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:31.777669 2560 patch_prober.go:28] interesting pod/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 29 14:29:31.777923 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:31.777741 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 14:29:33.269111 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:33.269074 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:29:34.778090 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.778025 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:29:34.834816 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.834776 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-podinfo\") pod \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " Jul 29 14:29:34.834816 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.834818 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-token\") pod \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " Jul 29 14:29:34.835075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.834841 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istiod-ca-cert\") pod \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " Jul 29 14:29:34.835075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.834937 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-credential-socket\") pod \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " Jul 29 14:29:34.835075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.834967 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-certs\") pod \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " Jul 29 14:29:34.835075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.834989 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-envoy\") pod \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " Jul 29 14:29:34.835075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.835039 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7gzk\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-kube-api-access-r7gzk\") pod \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " Jul 29 14:29:34.835336 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.835083 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-data\") pod \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " Jul 29 14:29:34.835336 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.835233 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5dd815cf-af4d-4a14-ac96-f162a7011d0d" (UID: "5dd815cf-af4d-4a14-ac96-f162a7011d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:29:34.835336 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.835277 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5dd815cf-af4d-4a14-ac96-f162a7011d0d" (UID: "5dd815cf-af4d-4a14-ac96-f162a7011d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:29:34.835336 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.835277 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5dd815cf-af4d-4a14-ac96-f162a7011d0d" (UID: "5dd815cf-af4d-4a14-ac96-f162a7011d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:29:34.835582 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.835508 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5dd815cf-af4d-4a14-ac96-f162a7011d0d" (UID: "5dd815cf-af4d-4a14-ac96-f162a7011d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:29:34.835711 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.835679 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-socket\") pod \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\" (UID: \"5dd815cf-af4d-4a14-ac96-f162a7011d0d\") " Jul 29 14:29:34.836280 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.836253 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-data" (OuterVolumeSpecName: "istio-data") pod "5dd815cf-af4d-4a14-ac96-f162a7011d0d" (UID: "5dd815cf-af4d-4a14-ac96-f162a7011d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:29:34.836400 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.836257 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:34.836400 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.836314 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istiod-ca-cert\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:34.836400 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.836333 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-credential-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:34.836400 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.836348 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-workload-certs\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:34.837252 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.837227 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5dd815cf-af4d-4a14-ac96-f162a7011d0d" (UID: "5dd815cf-af4d-4a14-ac96-f162a7011d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 14:29:34.837421 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.837398 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-token" (OuterVolumeSpecName: "istio-token") pod "5dd815cf-af4d-4a14-ac96-f162a7011d0d" (UID: "5dd815cf-af4d-4a14-ac96-f162a7011d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:29:34.837532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.837514 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5dd815cf-af4d-4a14-ac96-f162a7011d0d" (UID: "5dd815cf-af4d-4a14-ac96-f162a7011d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:29:34.837737 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.837720 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-kube-api-access-r7gzk" (OuterVolumeSpecName: "kube-api-access-r7gzk") pod "5dd815cf-af4d-4a14-ac96-f162a7011d0d" (UID: "5dd815cf-af4d-4a14-ac96-f162a7011d0d"). InnerVolumeSpecName "kube-api-access-r7gzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:29:34.937905 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.937871 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-podinfo\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:34.937905 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.937902 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-token\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:34.937905 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.937911 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-envoy\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:34.938153 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.937922 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7gzk\" (UniqueName: \"kubernetes.io/projected/5dd815cf-af4d-4a14-ac96-f162a7011d0d-kube-api-access-r7gzk\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:34.938153 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:34.937930 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd815cf-af4d-4a14-ac96-f162a7011d0d-istio-data\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:29:35.267623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.267590 2560 generic.go:356] "Generic (PLEG): container finished" podID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerID="e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09" exitCode=0 Jul 29 14:29:35.267823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.267649 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" Jul 29 14:29:35.267823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.267674 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" event={"ID":"5dd815cf-af4d-4a14-ac96-f162a7011d0d","Type":"ContainerDied","Data":"e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09"} Jul 29 14:29:35.267823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.267732 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48" event={"ID":"5dd815cf-af4d-4a14-ac96-f162a7011d0d","Type":"ContainerDied","Data":"8b36fc8786ca04c01817b75d39d9dd077911c52e248b3b149d1d6df3f76e8141"} Jul 29 14:29:35.267823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.267755 2560 scope.go:117] "RemoveContainer" containerID="e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09" Jul 29 14:29:35.278267 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.278246 2560 scope.go:117] "RemoveContainer" containerID="22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7" Jul 29 14:29:35.287494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.287446 2560 scope.go:117] "RemoveContainer" containerID="e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09" Jul 29 14:29:35.288174 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:29:35.288148 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09\": container with ID starting with e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09 not found: ID does not exist" containerID="e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09" Jul 29 14:29:35.288292 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.288184 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09"} err="failed to get container status \"e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09\": rpc error: code = NotFound desc = could not find container \"e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09\": container with ID starting with e40e41539081d9cf369085a349ca66022faf8adf9b520f199acff273e8745c09 not found: ID does not exist" Jul 29 14:29:35.288292 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.288220 2560 scope.go:117] "RemoveContainer" containerID="22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7" Jul 29 14:29:35.288594 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:29:35.288570 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7\": container with ID starting with 22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7 not found: ID does not exist" containerID="22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7" Jul 29 14:29:35.288666 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.288606 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7"} err="failed to get container status \"22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7\": rpc error: code = NotFound desc = could not find container \"22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7\": container with ID starting with 22517086b74d3081083095f50118674fafb0b1e73947ce85303153e5ef7140a7 not found: ID does not exist" Jul 29 14:29:35.290833 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.290810 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48"] Jul 29 14:29:35.294828 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.294801 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d8b7adbd-openshift-default-6c4b8d4448-kfq48"] Jul 29 14:29:35.381677 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:35.381639 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" path="/var/lib/kubelet/pods/5dd815cf-af4d-4a14-ac96-f162a7011d0d/volumes" Jul 29 14:29:43.848069 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:43.848027 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:29:58.167829 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.167782 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg"] Jul 29 14:29:58.168451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.168275 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2f6120-8569-4653-8662-c468259be16d" containerName="revoke-keys" Jul 29 14:29:58.168451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.168289 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2f6120-8569-4653-8662-c468259be16d" containerName="revoke-keys" Jul 29 14:29:58.168451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.168320 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" Jul 29 14:29:58.168451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.168325 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" Jul 29 14:29:58.168451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.168333 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" Jul 29 14:29:58.168451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.168338 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" Jul 29 14:29:58.168451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.168394 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" Jul 29 14:29:58.168451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.168402 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd815cf-af4d-4a14-ac96-f162a7011d0d" containerName="istio-proxy" Jul 29 14:29:58.168451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.168411 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2f6120-8569-4653-8662-c468259be16d" containerName="revoke-keys" Jul 29 14:29:58.171512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.171495 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" Jul 29 14:29:58.173920 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.173895 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:29:58.178915 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.178883 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg"] Jul 29 14:29:58.269170 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.269131 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cd90dc-5fa2-4739-a568-7c82c7f5cdf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg\" (UID: \"86cd90dc-5fa2-4739-a568-7c82c7f5cdf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" Jul 29 14:29:58.370284 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.370240 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cd90dc-5fa2-4739-a568-7c82c7f5cdf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg\" (UID: \"86cd90dc-5fa2-4739-a568-7c82c7f5cdf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" Jul 29 14:29:58.370925 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.370904 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cd90dc-5fa2-4739-a568-7c82c7f5cdf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg\" (UID: \"86cd90dc-5fa2-4739-a568-7c82c7f5cdf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" Jul 29 14:29:58.484569 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.484508 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" Jul 29 14:29:58.629279 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:58.629248 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg"] Jul 29 14:29:58.631272 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:29:58.631236 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cd90dc_5fa2_4739_a568_7c82c7f5cdf6.slice/crio-4089b96de1b51384743b027d8f28f1476aae3be3fefb1bd0f06518de3275b9d9 WatchSource:0}: Error finding container 4089b96de1b51384743b027d8f28f1476aae3be3fefb1bd0f06518de3275b9d9: Status 404 returned error can't find the container with id 4089b96de1b51384743b027d8f28f1476aae3be3fefb1bd0f06518de3275b9d9 Jul 29 14:29:59.417287 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:59.417248 2560 generic.go:356] "Generic (PLEG): container finished" podID="86cd90dc-5fa2-4739-a568-7c82c7f5cdf6" containerID="c48d1edb7c84fde69b5e2a2a8c75529733274d9002a8b3dc3beb373c16261a52" exitCode=0 Jul 29 14:29:59.417734 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:59.417301 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" event={"ID":"86cd90dc-5fa2-4739-a568-7c82c7f5cdf6","Type":"ContainerDied","Data":"c48d1edb7c84fde69b5e2a2a8c75529733274d9002a8b3dc3beb373c16261a52"} Jul 29 14:29:59.417734 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:29:59.417360 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" event={"ID":"86cd90dc-5fa2-4739-a568-7c82c7f5cdf6","Type":"ContainerStarted","Data":"4089b96de1b51384743b027d8f28f1476aae3be3fefb1bd0f06518de3275b9d9"} Jul 29 14:30:00.133691 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.133660 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn"] Jul 29 14:30:00.137360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.137342 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp"] Jul 29 14:30:00.137476 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.137456 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn" Jul 29 14:30:00.140971 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.140945 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp" Jul 29 14:30:00.145466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.144668 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn"] Jul 29 14:30:00.148579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.148299 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp"] Jul 29 14:30:00.152076 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.152049 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn" Jul 29 14:30:00.161158 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.161138 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp" Jul 29 14:30:00.310671 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.310643 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp"] Jul 29 14:30:00.313347 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:30:00.313313 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea360ac_5c03_4df1_ad14_b54d7d3900a9.slice/crio-ff19159b085b28db2cf593a0393bc184bbdfc1acb909dfc4109e5d871da56739 WatchSource:0}: Error finding container ff19159b085b28db2cf593a0393bc184bbdfc1acb909dfc4109e5d871da56739: Status 404 returned error can't find the container with id ff19159b085b28db2cf593a0393bc184bbdfc1acb909dfc4109e5d871da56739 Jul 29 14:30:00.334833 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.334805 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn"] Jul 29 14:30:00.336507 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:30:00.336476 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb203d13_bcb5_4386_8667_09166e8b6b48.slice/crio-92f3207e6fb65012f598a410774d1966a69de597b959b4e65e830128270e348d WatchSource:0}: Error finding container 92f3207e6fb65012f598a410774d1966a69de597b959b4e65e830128270e348d: Status 404 returned error can't find the container with id 92f3207e6fb65012f598a410774d1966a69de597b959b4e65e830128270e348d Jul 29 14:30:00.425111 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.425073 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp" event={"ID":"4ea360ac-5c03-4df1-ad14-b54d7d3900a9","Type":"ContainerStarted","Data":"ff19159b085b28db2cf593a0393bc184bbdfc1acb909dfc4109e5d871da56739"} Jul 29 14:30:00.426388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.426365 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn" event={"ID":"db203d13-bcb5-4386-8667-09166e8b6b48","Type":"ContainerStarted","Data":"92f3207e6fb65012f598a410774d1966a69de597b959b4e65e830128270e348d"} Jul 29 14:30:00.553979 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.553947 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" Jul 29 14:30:00.591196 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.591165 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cd90dc-5fa2-4739-a568-7c82c7f5cdf6-maas-api-service-ca\") pod \"86cd90dc-5fa2-4739-a568-7c82c7f5cdf6\" (UID: \"86cd90dc-5fa2-4739-a568-7c82c7f5cdf6\") " Jul 29 14:30:00.591643 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.591614 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cd90dc-5fa2-4739-a568-7c82c7f5cdf6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86cd90dc-5fa2-4739-a568-7c82c7f5cdf6" (UID: "86cd90dc-5fa2-4739-a568-7c82c7f5cdf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:30:00.692075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:00.692043 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cd90dc-5fa2-4739-a568-7c82c7f5cdf6-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:01.433959 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:01.433930 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" Jul 29 14:30:01.434386 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:01.433963 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg" event={"ID":"86cd90dc-5fa2-4739-a568-7c82c7f5cdf6","Type":"ContainerDied","Data":"4089b96de1b51384743b027d8f28f1476aae3be3fefb1bd0f06518de3275b9d9"} Jul 29 14:30:01.434386 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:01.434030 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4089b96de1b51384743b027d8f28f1476aae3be3fefb1bd0f06518de3275b9d9" Jul 29 14:30:01.435709 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:01.435686 2560 generic.go:356] "Generic (PLEG): container finished" podID="4ea360ac-5c03-4df1-ad14-b54d7d3900a9" containerID="2c37ce88f0586b842042d7b3ddc6e14970c7122c01117e69f50e80f5b60ce842" exitCode=0 Jul 29 14:30:01.435867 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:01.435719 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp" event={"ID":"4ea360ac-5c03-4df1-ad14-b54d7d3900a9","Type":"ContainerDied","Data":"2c37ce88f0586b842042d7b3ddc6e14970c7122c01117e69f50e80f5b60ce842"} Jul 29 14:30:01.437558 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:01.437519 2560 generic.go:356] "Generic (PLEG): container finished" podID="db203d13-bcb5-4386-8667-09166e8b6b48" containerID="650d330e77da2a7242866435e46036ae60a9553e5b3f74e9b59dc2bcaf6e5e3e" exitCode=0 Jul 29 14:30:01.437650 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:01.437567 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn" event={"ID":"db203d13-bcb5-4386-8667-09166e8b6b48","Type":"ContainerDied","Data":"650d330e77da2a7242866435e46036ae60a9553e5b3f74e9b59dc2bcaf6e5e3e"} Jul 29 14:30:02.601855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:02.601828 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp" Jul 29 14:30:02.604949 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:02.604928 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn" Jul 29 14:30:03.450166 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:03.450136 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp" Jul 29 14:30:03.450362 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:03.450177 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp" event={"ID":"4ea360ac-5c03-4df1-ad14-b54d7d3900a9","Type":"ContainerDied","Data":"ff19159b085b28db2cf593a0393bc184bbdfc1acb909dfc4109e5d871da56739"} Jul 29 14:30:03.450362 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:03.450231 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff19159b085b28db2cf593a0393bc184bbdfc1acb909dfc4109e5d871da56739" Jul 29 14:30:03.451676 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:03.451647 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn" Jul 29 14:30:03.451816 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:03.451693 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755590-9mlqn" event={"ID":"db203d13-bcb5-4386-8667-09166e8b6b48","Type":"ContainerDied","Data":"92f3207e6fb65012f598a410774d1966a69de597b959b4e65e830128270e348d"} Jul 29 14:30:03.451816 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:03.451733 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f3207e6fb65012f598a410774d1966a69de597b959b4e65e830128270e348d" Jul 29 14:30:03.905143 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:03.905068 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp"] Jul 29 14:30:03.907932 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:03.907904 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-59408d2e-29755590-7dxgp"] Jul 29 14:30:04.107687 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:04.107649 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg"] Jul 29 14:30:04.110668 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:04.110636 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59408d2e-8t9xg"] Jul 29 14:30:05.027970 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.027937 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js"] Jul 29 14:30:05.028399 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.028385 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea360ac-5c03-4df1-ad14-b54d7d3900a9" containerName="cleanup" Jul 29 14:30:05.028464 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.028401 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea360ac-5c03-4df1-ad14-b54d7d3900a9" containerName="cleanup" Jul 29 14:30:05.028464 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.028419 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db203d13-bcb5-4386-8667-09166e8b6b48" containerName="cleanup" Jul 29 14:30:05.028464 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.028425 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="db203d13-bcb5-4386-8667-09166e8b6b48" containerName="cleanup" Jul 29 14:30:05.028464 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.028447 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86cd90dc-5fa2-4739-a568-7c82c7f5cdf6" containerName="revoke-keys" Jul 29 14:30:05.028464 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.028452 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="86cd90dc-5fa2-4739-a568-7c82c7f5cdf6" containerName="revoke-keys" Jul 29 14:30:05.028646 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.028515 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea360ac-5c03-4df1-ad14-b54d7d3900a9" containerName="cleanup" Jul 29 14:30:05.028646 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.028524 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="db203d13-bcb5-4386-8667-09166e8b6b48" containerName="cleanup" Jul 29 14:30:05.028646 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.028531 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="86cd90dc-5fa2-4739-a568-7c82c7f5cdf6" containerName="revoke-keys" Jul 29 14:30:05.031839 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.031813 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.034685 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.034663 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ee08ac7d-openshift-default-dockercfg-b9xx2\"" Jul 29 14:30:05.049744 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.049713 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js"] Jul 29 14:30:05.132185 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.132147 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8nt\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-kube-api-access-zq8nt\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.132367 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.132207 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-socket\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.132367 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.132247 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-data\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.132367 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.132279 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-credential-socket\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.132578 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.132396 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-istio-token\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.132578 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.132469 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-envoy\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.132578 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.132508 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-certs\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.132578 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.132534 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03718e81-e25a-4e96-aca9-4107415e8a68-istiod-ca-cert\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.132800 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.132579 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03718e81-e25a-4e96-aca9-4107415e8a68-istio-podinfo\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233176 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233133 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-socket\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233184 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-data\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-credential-socket\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233240 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-istio-token\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233271 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-envoy\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233294 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-certs\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233312 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03718e81-e25a-4e96-aca9-4107415e8a68-istiod-ca-cert\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233338 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03718e81-e25a-4e96-aca9-4107415e8a68-istio-podinfo\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233374 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8nt\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-kube-api-access-zq8nt\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233901 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233712 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-socket\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233901 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233710 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-credential-socket\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.233901 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.233874 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-certs\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.234052 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.234026 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-data\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.234160 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.234111 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03718e81-e25a-4e96-aca9-4107415e8a68-istiod-ca-cert\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.236485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.236460 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-envoy\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.236761 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.236736 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03718e81-e25a-4e96-aca9-4107415e8a68-istio-podinfo\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.241889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.241862 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-istio-token\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.242006 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.241932 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8nt\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-kube-api-access-zq8nt\") pod \"e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.343359 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.343262 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:05.384798 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.384758 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea360ac-5c03-4df1-ad14-b54d7d3900a9" path="/var/lib/kubelet/pods/4ea360ac-5c03-4df1-ad14-b54d7d3900a9/volumes" Jul 29 14:30:05.385150 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.385135 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86cd90dc-5fa2-4739-a568-7c82c7f5cdf6" path="/var/lib/kubelet/pods/86cd90dc-5fa2-4739-a568-7c82c7f5cdf6/volumes" Jul 29 14:30:05.500591 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.500565 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js"] Jul 29 14:30:05.502986 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:30:05.502954 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03718e81_e25a_4e96_aca9_4107415e8a68.slice/crio-bd6e6e016bebdc79e154737cb121b613ad01f6ea55a57a96ad5cf2fcf5ead2ad WatchSource:0}: Error finding container bd6e6e016bebdc79e154737cb121b613ad01f6ea55a57a96ad5cf2fcf5ead2ad: Status 404 returned error can't find the container with id bd6e6e016bebdc79e154737cb121b613ad01f6ea55a57a96ad5cf2fcf5ead2ad Jul 29 14:30:05.505457 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.505414 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:30:05.505611 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.505512 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:30:05.505611 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:05.505577 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:30:06.476370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:06.476325 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" event={"ID":"03718e81-e25a-4e96-aca9-4107415e8a68","Type":"ContainerStarted","Data":"2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d"} Jul 29 14:30:06.476809 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:06.476399 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" event={"ID":"03718e81-e25a-4e96-aca9-4107415e8a68","Type":"ContainerStarted","Data":"bd6e6e016bebdc79e154737cb121b613ad01f6ea55a57a96ad5cf2fcf5ead2ad"} Jul 29 14:30:06.500318 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:06.500254 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" podStartSLOduration=1.500229987 podStartE2EDuration="1.500229987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:30:06.49621573 +0000 UTC m=+1739.697716903" watchObservedRunningTime="2026-07-29 14:30:06.500229987 +0000 UTC m=+1739.701731160" Jul 29 14:30:07.344314 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:07.344268 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:07.349373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:07.349344 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:07.482823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:07.482787 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:07.484004 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:07.483985 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:08.105231 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:08.105195 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:30:20.249748 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:20.249703 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:30:22.538850 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:22.537850 2560 patch_prober.go:28] interesting pod/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:53850->10.132.0.64:15021: read: connection reset by peer" start-of-body= Jul 29 14:30:22.538850 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:22.537937 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:53850->10.132.0.64:15021: read: connection reset by peer" Jul 29 14:30:22.587608 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:22.587574 2560 generic.go:356] "Generic (PLEG): container finished" podID="03718e81-e25a-4e96-aca9-4107415e8a68" containerID="2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d" exitCode=0 Jul 29 14:30:22.587608 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:22.587576 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" event={"ID":"03718e81-e25a-4e96-aca9-4107415e8a68","Type":"ContainerDied","Data":"2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d"} Jul 29 14:30:22.588120 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:22.588105 2560 scope.go:117] "RemoveContainer" containerID="2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d" Jul 29 14:30:22.589328 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:22.589295 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:30:22.589443 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:22.589384 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:30:22.589443 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:22.589429 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:30:23.597469 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:23.597433 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" event={"ID":"03718e81-e25a-4e96-aca9-4107415e8a68","Type":"ContainerStarted","Data":"b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924"} Jul 29 14:30:24.344407 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:24.344368 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:24.344763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:24.344733 2560 patch_prober.go:28] interesting pod/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 14:30:24.344840 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:24.344795 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 14:30:25.344459 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:25.344418 2560 patch_prober.go:28] interesting pod/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 14:30:25.344941 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:25.344498 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 14:30:26.344608 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:26.344529 2560 patch_prober.go:28] interesting pod/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 14:30:26.345038 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:26.344633 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 14:30:27.344239 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:27.344196 2560 patch_prober.go:28] interesting pod/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 14:30:27.344435 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:27.344263 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 14:30:28.344073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:28.344029 2560 patch_prober.go:28] interesting pod/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 14:30:28.344660 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:28.344100 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 14:30:29.344154 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:29.344113 2560 patch_prober.go:28] interesting pod/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 14:30:29.344702 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:29.344172 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 14:30:30.348106 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:30.348076 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:30.348586 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:30.348344 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:30.348978 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:30.348960 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:33.657139 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:33.657097 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv"] Jul 29 14:30:33.661983 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:33.661959 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" Jul 29 14:30:33.664697 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:33.664671 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:30:33.667936 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:33.667911 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv"] Jul 29 14:30:33.819956 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:33.819913 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d72a8-b260-4f22-866f-0e8e176607bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv\" (UID: \"aa7d72a8-b260-4f22-866f-0e8e176607bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" Jul 29 14:30:33.921347 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:33.921243 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d72a8-b260-4f22-866f-0e8e176607bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv\" (UID: \"aa7d72a8-b260-4f22-866f-0e8e176607bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" Jul 29 14:30:33.921923 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:33.921902 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d72a8-b260-4f22-866f-0e8e176607bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv\" (UID: \"aa7d72a8-b260-4f22-866f-0e8e176607bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" Jul 29 14:30:33.976419 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:33.976374 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" Jul 29 14:30:34.124737 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:34.124711 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv"] Jul 29 14:30:34.126696 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:30:34.126668 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7d72a8_b260_4f22_866f_0e8e176607bc.slice/crio-785cfdd4169f79d72b4f7c30a49b59581e3f2a1e87b6d6963b75d392cc3c2f56 WatchSource:0}: Error finding container 785cfdd4169f79d72b4f7c30a49b59581e3f2a1e87b6d6963b75d392cc3c2f56: Status 404 returned error can't find the container with id 785cfdd4169f79d72b4f7c30a49b59581e3f2a1e87b6d6963b75d392cc3c2f56 Jul 29 14:30:34.669729 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:34.669634 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" event={"ID":"aa7d72a8-b260-4f22-866f-0e8e176607bc","Type":"ContainerStarted","Data":"c915b3236e3a61fde75cc5260b5b62999ba26158ad8276fbcbfe68239ed6cbaf"} Jul 29 14:30:34.669729 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:34.669697 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" event={"ID":"aa7d72a8-b260-4f22-866f-0e8e176607bc","Type":"ContainerStarted","Data":"785cfdd4169f79d72b4f7c30a49b59581e3f2a1e87b6d6963b75d392cc3c2f56"} Jul 29 14:30:34.687011 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:34.686952 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" podStartSLOduration=1.686933926 podStartE2EDuration="1.686933926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:30:34.685750858 +0000 UTC m=+1767.887252055" watchObservedRunningTime="2026-07-29 14:30:34.686933926 +0000 UTC m=+1767.888435144" Jul 29 14:30:35.677252 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:35.677212 2560 generic.go:356] "Generic (PLEG): container finished" podID="aa7d72a8-b260-4f22-866f-0e8e176607bc" containerID="c915b3236e3a61fde75cc5260b5b62999ba26158ad8276fbcbfe68239ed6cbaf" exitCode=0 Jul 29 14:30:35.677690 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:35.677297 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" event={"ID":"aa7d72a8-b260-4f22-866f-0e8e176607bc","Type":"ContainerDied","Data":"c915b3236e3a61fde75cc5260b5b62999ba26158ad8276fbcbfe68239ed6cbaf"} Jul 29 14:30:36.826474 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:36.826448 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" Jul 29 14:30:36.951194 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:36.951095 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d72a8-b260-4f22-866f-0e8e176607bc-maas-api-service-ca\") pod \"aa7d72a8-b260-4f22-866f-0e8e176607bc\" (UID: \"aa7d72a8-b260-4f22-866f-0e8e176607bc\") " Jul 29 14:30:36.951433 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:36.951410 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7d72a8-b260-4f22-866f-0e8e176607bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa7d72a8-b260-4f22-866f-0e8e176607bc" (UID: "aa7d72a8-b260-4f22-866f-0e8e176607bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:30:37.052357 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:37.052317 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d72a8-b260-4f22-866f-0e8e176607bc-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:37.690267 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:37.690235 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" Jul 29 14:30:37.690464 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:37.690234 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv" event={"ID":"aa7d72a8-b260-4f22-866f-0e8e176607bc","Type":"ContainerDied","Data":"785cfdd4169f79d72b4f7c30a49b59581e3f2a1e87b6d6963b75d392cc3c2f56"} Jul 29 14:30:37.690464 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:37.690365 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="785cfdd4169f79d72b4f7c30a49b59581e3f2a1e87b6d6963b75d392cc3c2f56" Jul 29 14:30:38.776144 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:38.776104 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv"] Jul 29 14:30:38.780274 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:38.780246 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee08ac7d-7frzv"] Jul 29 14:30:39.381103 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:39.381069 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7d72a8-b260-4f22-866f-0e8e176607bc" path="/var/lib/kubelet/pods/aa7d72a8-b260-4f22-866f-0e8e176607bc/volumes" Jul 29 14:30:39.935262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:39.935216 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js"] Jul 29 14:30:39.935801 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:39.935502 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" containerID="cri-o://b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924" gracePeriod=30 Jul 29 14:30:42.754801 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:42.754762 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:30:45.203674 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.203591 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:45.236295 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236263 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-data\") pod \"03718e81-e25a-4e96-aca9-4107415e8a68\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " Jul 29 14:30:45.236494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236328 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-credential-socket\") pod \"03718e81-e25a-4e96-aca9-4107415e8a68\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " Jul 29 14:30:45.236494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236361 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq8nt\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-kube-api-access-zq8nt\") pod \"03718e81-e25a-4e96-aca9-4107415e8a68\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " Jul 29 14:30:45.236494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236388 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-certs\") pod \"03718e81-e25a-4e96-aca9-4107415e8a68\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " Jul 29 14:30:45.236494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236444 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03718e81-e25a-4e96-aca9-4107415e8a68-istiod-ca-cert\") pod \"03718e81-e25a-4e96-aca9-4107415e8a68\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " Jul 29 14:30:45.236494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236472 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03718e81-e25a-4e96-aca9-4107415e8a68-istio-podinfo\") pod \"03718e81-e25a-4e96-aca9-4107415e8a68\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " Jul 29 14:30:45.236795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236526 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-socket\") pod \"03718e81-e25a-4e96-aca9-4107415e8a68\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " Jul 29 14:30:45.236795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236585 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-istio-token\") pod \"03718e81-e25a-4e96-aca9-4107415e8a68\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " Jul 29 14:30:45.236795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236734 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03718e81-e25a-4e96-aca9-4107415e8a68" (UID: "03718e81-e25a-4e96-aca9-4107415e8a68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:30:45.236916 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236874 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03718e81-e25a-4e96-aca9-4107415e8a68-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03718e81-e25a-4e96-aca9-4107415e8a68" (UID: "03718e81-e25a-4e96-aca9-4107415e8a68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:30:45.237036 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.236989 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-credential-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:45.237036 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.237012 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03718e81-e25a-4e96-aca9-4107415e8a68-istiod-ca-cert\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:45.237036 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.237122 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03718e81-e25a-4e96-aca9-4107415e8a68" (UID: "03718e81-e25a-4e96-aca9-4107415e8a68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:30:45.237600 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.237140 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03718e81-e25a-4e96-aca9-4107415e8a68" (UID: "03718e81-e25a-4e96-aca9-4107415e8a68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:30:45.237600 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.237414 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-data" (OuterVolumeSpecName: "istio-data") pod "03718e81-e25a-4e96-aca9-4107415e8a68" (UID: "03718e81-e25a-4e96-aca9-4107415e8a68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:30:45.238838 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.238810 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-kube-api-access-zq8nt" (OuterVolumeSpecName: "kube-api-access-zq8nt") pod "03718e81-e25a-4e96-aca9-4107415e8a68" (UID: "03718e81-e25a-4e96-aca9-4107415e8a68"). InnerVolumeSpecName "kube-api-access-zq8nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:30:45.238982 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.238962 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03718e81-e25a-4e96-aca9-4107415e8a68-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03718e81-e25a-4e96-aca9-4107415e8a68" (UID: "03718e81-e25a-4e96-aca9-4107415e8a68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 14:30:45.239275 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.239255 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-istio-token" (OuterVolumeSpecName: "istio-token") pod "03718e81-e25a-4e96-aca9-4107415e8a68" (UID: "03718e81-e25a-4e96-aca9-4107415e8a68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:30:45.338005 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.337970 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-envoy\") pod \"03718e81-e25a-4e96-aca9-4107415e8a68\" (UID: \"03718e81-e25a-4e96-aca9-4107415e8a68\") " Jul 29 14:30:45.338201 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.338189 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq8nt\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-kube-api-access-zq8nt\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:45.338247 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.338204 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-certs\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:45.338247 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.338215 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03718e81-e25a-4e96-aca9-4107415e8a68-istio-podinfo\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:45.338247 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.338223 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-workload-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:45.338247 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.338232 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03718e81-e25a-4e96-aca9-4107415e8a68-istio-token\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:45.338247 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.338240 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-data\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:45.340272 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.340246 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03718e81-e25a-4e96-aca9-4107415e8a68" (UID: "03718e81-e25a-4e96-aca9-4107415e8a68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:30:45.439251 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.439208 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03718e81-e25a-4e96-aca9-4107415e8a68-istio-envoy\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:30:45.741698 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.741656 2560 generic.go:356] "Generic (PLEG): container finished" podID="03718e81-e25a-4e96-aca9-4107415e8a68" containerID="b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924" exitCode=0 Jul 29 14:30:45.741905 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.741713 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" Jul 29 14:30:45.741905 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.741746 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" event={"ID":"03718e81-e25a-4e96-aca9-4107415e8a68","Type":"ContainerDied","Data":"b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924"} Jul 29 14:30:45.741905 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.741805 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js" event={"ID":"03718e81-e25a-4e96-aca9-4107415e8a68","Type":"ContainerDied","Data":"bd6e6e016bebdc79e154737cb121b613ad01f6ea55a57a96ad5cf2fcf5ead2ad"} Jul 29 14:30:45.741905 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.741836 2560 scope.go:117] "RemoveContainer" containerID="b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924" Jul 29 14:30:45.752064 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.752036 2560 scope.go:117] "RemoveContainer" containerID="2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d" Jul 29 14:30:45.760220 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.760185 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js"] Jul 29 14:30:45.763509 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.763489 2560 scope.go:117] "RemoveContainer" containerID="b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924" Jul 29 14:30:45.763884 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:30:45.763863 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924\": container with ID starting with b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924 not found: ID does not exist" containerID="b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924" Jul 29 14:30:45.763930 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.763894 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924"} err="failed to get container status \"b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924\": rpc error: code = NotFound desc = could not find container \"b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924\": container with ID starting with b53710947b98d8a8e1eb7205f9def8f0716f6745fd558d59d6578b7c63812924 not found: ID does not exist" Jul 29 14:30:45.763930 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.763914 2560 scope.go:117] "RemoveContainer" containerID="2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d" Jul 29 14:30:45.764118 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.764097 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ee08ac7d-openshift-default-7bb9848494-lg5js"] Jul 29 14:30:45.764188 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:30:45.764166 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d\": container with ID starting with 2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d not found: ID does not exist" containerID="2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d" Jul 29 14:30:45.764226 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:45.764196 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d"} err="failed to get container status \"2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d\": rpc error: code = NotFound desc = could not find container \"2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d\": container with ID starting with 2d76fd0c656d5ace022e29452ef78ea2bf7435cef6a3fcec530bb7628406da7d not found: ID does not exist" Jul 29 14:30:47.381984 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:47.381950 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" path="/var/lib/kubelet/pods/03718e81-e25a-4e96-aca9-4107415e8a68/volumes" Jul 29 14:30:53.248440 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:30:53.248399 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:31:07.512740 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:07.512710 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:31:07.516195 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:07.516171 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:31:07.528350 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:07.528323 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:31:07.531584 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:07.531562 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:31:08.240014 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.239973 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf"] Jul 29 14:31:08.240509 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.240480 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" Jul 29 14:31:08.240509 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.240502 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" Jul 29 14:31:08.240704 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.240585 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7d72a8-b260-4f22-866f-0e8e176607bc" containerName="revoke-keys" Jul 29 14:31:08.240704 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.240595 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7d72a8-b260-4f22-866f-0e8e176607bc" containerName="revoke-keys" Jul 29 14:31:08.240813 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.240713 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" Jul 29 14:31:08.240813 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.240726 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" Jul 29 14:31:08.240813 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.240737 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7d72a8-b260-4f22-866f-0e8e176607bc" containerName="revoke-keys" Jul 29 14:31:08.243955 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.243931 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" Jul 29 14:31:08.246747 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.246727 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:31:08.251718 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.251685 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf"] Jul 29 14:31:08.356893 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.356854 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e304c723-621e-4b91-9587-2dd6b06f49c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-330348dc-xj6pf\" (UID: \"e304c723-621e-4b91-9587-2dd6b06f49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" Jul 29 14:31:08.458468 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.458424 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e304c723-621e-4b91-9587-2dd6b06f49c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-330348dc-xj6pf\" (UID: \"e304c723-621e-4b91-9587-2dd6b06f49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" Jul 29 14:31:08.459157 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.459134 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e304c723-621e-4b91-9587-2dd6b06f49c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-330348dc-xj6pf\" (UID: \"e304c723-621e-4b91-9587-2dd6b06f49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" Jul 29 14:31:08.557168 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.557073 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" Jul 29 14:31:08.706927 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.706893 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf"] Jul 29 14:31:08.709824 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:31:08.709790 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode304c723_621e_4b91_9587_2dd6b06f49c6.slice/crio-66412223e754a9c7d2064bfdd03b59e1b7d970e9026b0443ffa78bc36cab241a WatchSource:0}: Error finding container 66412223e754a9c7d2064bfdd03b59e1b7d970e9026b0443ffa78bc36cab241a: Status 404 returned error can't find the container with id 66412223e754a9c7d2064bfdd03b59e1b7d970e9026b0443ffa78bc36cab241a Jul 29 14:31:08.893160 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:08.893074 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" event={"ID":"e304c723-621e-4b91-9587-2dd6b06f49c6","Type":"ContainerStarted","Data":"66412223e754a9c7d2064bfdd03b59e1b7d970e9026b0443ffa78bc36cab241a"} Jul 29 14:31:09.901480 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:09.901388 2560 generic.go:356] "Generic (PLEG): container finished" podID="e304c723-621e-4b91-9587-2dd6b06f49c6" containerID="7a26fdbc858a4be30c56c261f83cba1a4723b44bdd50360b36b74d246f026156" exitCode=0 Jul 29 14:31:09.901891 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:09.901477 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" event={"ID":"e304c723-621e-4b91-9587-2dd6b06f49c6","Type":"ContainerDied","Data":"7a26fdbc858a4be30c56c261f83cba1a4723b44bdd50360b36b74d246f026156"} Jul 29 14:31:11.056481 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:11.056449 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" Jul 29 14:31:11.187449 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:11.187406 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e304c723-621e-4b91-9587-2dd6b06f49c6-maas-api-service-ca\") pod \"e304c723-621e-4b91-9587-2dd6b06f49c6\" (UID: \"e304c723-621e-4b91-9587-2dd6b06f49c6\") " Jul 29 14:31:11.187896 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:11.187869 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e304c723-621e-4b91-9587-2dd6b06f49c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e304c723-621e-4b91-9587-2dd6b06f49c6" (UID: "e304c723-621e-4b91-9587-2dd6b06f49c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:31:11.288869 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:11.288828 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e304c723-621e-4b91-9587-2dd6b06f49c6-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:31:11.916133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:11.916108 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" Jul 29 14:31:11.916314 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:11.916143 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf" event={"ID":"e304c723-621e-4b91-9587-2dd6b06f49c6","Type":"ContainerDied","Data":"66412223e754a9c7d2064bfdd03b59e1b7d970e9026b0443ffa78bc36cab241a"} Jul 29 14:31:11.916314 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:11.916193 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66412223e754a9c7d2064bfdd03b59e1b7d970e9026b0443ffa78bc36cab241a" Jul 29 14:31:14.404405 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:14.404373 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf"] Jul 29 14:31:14.407398 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:14.407368 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-330348dc-xj6pf"] Jul 29 14:31:15.380968 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:15.380933 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e304c723-621e-4b91-9587-2dd6b06f49c6" path="/var/lib/kubelet/pods/e304c723-621e-4b91-9587-2dd6b06f49c6/volumes" Jul 29 14:31:18.153916 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:31:18.153882 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:32:41.351483 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:41.351399 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:32:50.377011 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:50.376970 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:32:57.364700 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.364666 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8"] Jul 29 14:32:57.365145 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.365128 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" Jul 29 14:32:57.365202 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.365149 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="03718e81-e25a-4e96-aca9-4107415e8a68" containerName="istio-proxy" Jul 29 14:32:57.365202 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.365173 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e304c723-621e-4b91-9587-2dd6b06f49c6" containerName="revoke-keys" Jul 29 14:32:57.365202 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.365182 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e304c723-621e-4b91-9587-2dd6b06f49c6" containerName="revoke-keys" Jul 29 14:32:57.365346 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.365314 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e304c723-621e-4b91-9587-2dd6b06f49c6" containerName="revoke-keys" Jul 29 14:32:57.368591 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.368570 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.371338 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.371315 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-10f19eac-openshift-default-dockercfg-m6xgn\"" Jul 29 14:32:57.390458 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.390421 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8"] Jul 29 14:32:57.517889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.517852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-podinfo\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.517889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.517887 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-token\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.518116 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.517908 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nsnd\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-kube-api-access-5nsnd\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.518116 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.518000 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-data\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.518116 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.518050 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-certs\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.518116 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.518072 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-socket\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.518268 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.518130 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istiod-ca-cert\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.518268 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.518193 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-credential-socket\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.518268 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.518213 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-envoy\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.618990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.618889 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-certs\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.618990 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.618937 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-socket\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619014 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istiod-ca-cert\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619088 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-credential-socket\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619117 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-envoy\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619145 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-podinfo\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619167 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-token\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619216 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nsnd\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-kube-api-access-5nsnd\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619570 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619260 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-data\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619570 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619337 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-certs\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-socket\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.619932 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.619907 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istiod-ca-cert\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.620115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.620063 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-credential-socket\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.620115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.620071 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-data\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.621603 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.621582 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-envoy\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.621943 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.621921 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-podinfo\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.628011 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.627990 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-token\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.628458 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.628435 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nsnd\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-kube-api-access-5nsnd\") pod \"e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.680258 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.680219 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:57.837780 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.837750 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8"] Jul 29 14:32:57.840003 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:32:57.839973 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd60ecd_ac12_4d36_88f1_c687bdf4aaeb.slice/crio-1ad426cebc75abc08b2c8061e5682a47605cc50dfcf455f02a73f46e3345ab35 WatchSource:0}: Error finding container 1ad426cebc75abc08b2c8061e5682a47605cc50dfcf455f02a73f46e3345ab35: Status 404 returned error can't find the container with id 1ad426cebc75abc08b2c8061e5682a47605cc50dfcf455f02a73f46e3345ab35 Jul 29 14:32:57.842290 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.842253 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:32:57.842381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.842324 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:32:57.842381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:57.842359 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:32:58.594086 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:58.594050 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" event={"ID":"add60ecd-ac12-4d36-88f1-c687bdf4aaeb","Type":"ContainerStarted","Data":"d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3"} Jul 29 14:32:58.594585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:58.594103 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" event={"ID":"add60ecd-ac12-4d36-88f1-c687bdf4aaeb","Type":"ContainerStarted","Data":"1ad426cebc75abc08b2c8061e5682a47605cc50dfcf455f02a73f46e3345ab35"} Jul 29 14:32:58.617890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:58.617828 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" podStartSLOduration=1.617809695 podStartE2EDuration="1.617809695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:32:58.616509994 +0000 UTC m=+1911.818011165" watchObservedRunningTime="2026-07-29 14:32:58.617809695 +0000 UTC m=+1911.819310866" Jul 29 14:32:58.680881 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:58.680833 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:58.686435 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:58.686408 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:59.600508 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:59.600474 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:32:59.601627 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:32:59.601604 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:33:04.350972 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:04.350935 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:33:12.698369 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:12.698254 2560 generic.go:356] "Generic (PLEG): container finished" podID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerID="d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3" exitCode=0 Jul 29 14:33:12.698369 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:12.698327 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" event={"ID":"add60ecd-ac12-4d36-88f1-c687bdf4aaeb","Type":"ContainerDied","Data":"d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3"} Jul 29 14:33:12.698996 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:12.698745 2560 scope.go:117] "RemoveContainer" containerID="d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3" Jul 29 14:33:12.699965 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:12.699932 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:33:12.700084 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:12.700007 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:33:12.700084 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:12.700039 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:33:13.707736 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:13.707699 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" event={"ID":"add60ecd-ac12-4d36-88f1-c687bdf4aaeb","Type":"ContainerStarted","Data":"f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea"} Jul 29 14:33:14.601183 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:14.601087 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:33:14.680484 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:14.680437 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:33:14.681643 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:14.681604 2560 patch_prober.go:28] interesting pod/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 14:33:14.681797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:14.681678 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 14:33:15.681179 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:15.681146 2560 patch_prober.go:28] interesting pod/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 14:33:15.681635 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:15.681212 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 14:33:16.681594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:16.681524 2560 patch_prober.go:28] interesting pod/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 14:33:16.682002 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:16.681618 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 14:33:17.681061 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:17.681020 2560 patch_prober.go:28] interesting pod/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 14:33:17.681260 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:17.681098 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 14:33:18.685295 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:18.685264 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:33:18.686226 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:18.686208 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:33:32.817572 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:32.817521 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2"] Jul 29 14:33:32.820307 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:32.820290 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" Jul 29 14:33:32.822758 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:32.822735 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:33:32.829512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:32.829485 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2"] Jul 29 14:33:32.992886 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:32.992839 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e36740-87ac-4f5b-ab54-9a8e11add411-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2\" (UID: \"46e36740-87ac-4f5b-ab54-9a8e11add411\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" Jul 29 14:33:33.093478 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:33.093384 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e36740-87ac-4f5b-ab54-9a8e11add411-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2\" (UID: \"46e36740-87ac-4f5b-ab54-9a8e11add411\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" Jul 29 14:33:33.094092 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:33.094070 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e36740-87ac-4f5b-ab54-9a8e11add411-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2\" (UID: \"46e36740-87ac-4f5b-ab54-9a8e11add411\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" Jul 29 14:33:33.131654 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:33.131621 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" Jul 29 14:33:33.300578 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:33.300530 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2"] Jul 29 14:33:33.302494 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:33:33.302453 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e36740_87ac_4f5b_ab54_9a8e11add411.slice/crio-ce69578859fdd1ff5a4b1feca38ec5a960c7e9ca3fc1db97d3fd1fe9c5d93c93 WatchSource:0}: Error finding container ce69578859fdd1ff5a4b1feca38ec5a960c7e9ca3fc1db97d3fd1fe9c5d93c93: Status 404 returned error can't find the container with id ce69578859fdd1ff5a4b1feca38ec5a960c7e9ca3fc1db97d3fd1fe9c5d93c93 Jul 29 14:33:33.851663 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:33.851567 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" event={"ID":"46e36740-87ac-4f5b-ab54-9a8e11add411","Type":"ContainerStarted","Data":"ba9b3cee6885c2a9e886a29e466b966a095af2007ad6914e1758ee6ab5746083"} Jul 29 14:33:33.851663 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:33.851633 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" event={"ID":"46e36740-87ac-4f5b-ab54-9a8e11add411","Type":"ContainerStarted","Data":"ce69578859fdd1ff5a4b1feca38ec5a960c7e9ca3fc1db97d3fd1fe9c5d93c93"} Jul 29 14:33:33.869930 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:33.869873 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" podStartSLOduration=1.8698556769999999 podStartE2EDuration="1.869855677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:33:33.867132625 +0000 UTC m=+1947.068633797" watchObservedRunningTime="2026-07-29 14:33:33.869855677 +0000 UTC m=+1947.071356892" Jul 29 14:33:34.860204 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:34.860168 2560 generic.go:356] "Generic (PLEG): container finished" podID="46e36740-87ac-4f5b-ab54-9a8e11add411" containerID="ba9b3cee6885c2a9e886a29e466b966a095af2007ad6914e1758ee6ab5746083" exitCode=0 Jul 29 14:33:34.860622 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:34.860259 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" event={"ID":"46e36740-87ac-4f5b-ab54-9a8e11add411","Type":"ContainerDied","Data":"ba9b3cee6885c2a9e886a29e466b966a095af2007ad6914e1758ee6ab5746083"} Jul 29 14:33:36.008612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:36.008586 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" Jul 29 14:33:36.121850 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:36.121754 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e36740-87ac-4f5b-ab54-9a8e11add411-maas-api-service-ca\") pod \"46e36740-87ac-4f5b-ab54-9a8e11add411\" (UID: \"46e36740-87ac-4f5b-ab54-9a8e11add411\") " Jul 29 14:33:36.122140 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:36.122111 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e36740-87ac-4f5b-ab54-9a8e11add411-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46e36740-87ac-4f5b-ab54-9a8e11add411" (UID: "46e36740-87ac-4f5b-ab54-9a8e11add411"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:33:36.222776 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:36.222742 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e36740-87ac-4f5b-ab54-9a8e11add411-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:36.873617 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:36.873580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" event={"ID":"46e36740-87ac-4f5b-ab54-9a8e11add411","Type":"ContainerDied","Data":"ce69578859fdd1ff5a4b1feca38ec5a960c7e9ca3fc1db97d3fd1fe9c5d93c93"} Jul 29 14:33:36.873802 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:36.873636 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce69578859fdd1ff5a4b1feca38ec5a960c7e9ca3fc1db97d3fd1fe9c5d93c93" Jul 29 14:33:36.873802 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:36.873592 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2" Jul 29 14:33:43.591413 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:43.591370 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2"] Jul 29 14:33:43.595564 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:43.595518 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10f19eac-fb6l2"] Jul 29 14:33:45.381376 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:45.381336 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46e36740-87ac-4f5b-ab54-9a8e11add411" path="/var/lib/kubelet/pods/46e36740-87ac-4f5b-ab54-9a8e11add411/volumes" Jul 29 14:33:46.449112 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:46.449062 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:33:50.065215 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:50.065178 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8"] Jul 29 14:33:50.065741 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:50.065512 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" containerID="cri-o://f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea" gracePeriod=30 Jul 29 14:33:55.327558 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.327516 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:33:55.401750 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.401658 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-token\") pod \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " Jul 29 14:33:55.401750 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.401700 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-socket\") pod \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " Jul 29 14:33:55.401750 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.401726 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-envoy\") pod \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " Jul 29 14:33:55.402033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.401759 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istiod-ca-cert\") pod \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " Jul 29 14:33:55.402033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.401807 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nsnd\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-kube-api-access-5nsnd\") pod \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " Jul 29 14:33:55.402033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.401881 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-podinfo\") pod \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " Jul 29 14:33:55.402033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.401967 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-data\") pod \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " Jul 29 14:33:55.402033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.401998 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-certs\") pod \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " Jul 29 14:33:55.402033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.402023 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-credential-socket\") pod \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\" (UID: \"add60ecd-ac12-4d36-88f1-c687bdf4aaeb\") " Jul 29 14:33:55.402406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.402001 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "add60ecd-ac12-4d36-88f1-c687bdf4aaeb" (UID: "add60ecd-ac12-4d36-88f1-c687bdf4aaeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:33:55.402406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.402230 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "add60ecd-ac12-4d36-88f1-c687bdf4aaeb" (UID: "add60ecd-ac12-4d36-88f1-c687bdf4aaeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:33:55.402406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.402270 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "add60ecd-ac12-4d36-88f1-c687bdf4aaeb" (UID: "add60ecd-ac12-4d36-88f1-c687bdf4aaeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:33:55.402406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.402369 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:55.402406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.402389 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istiod-ca-cert\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:55.402406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.402399 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-workload-certs\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:55.402956 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.402425 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "add60ecd-ac12-4d36-88f1-c687bdf4aaeb" (UID: "add60ecd-ac12-4d36-88f1-c687bdf4aaeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:33:55.403145 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.403099 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "add60ecd-ac12-4d36-88f1-c687bdf4aaeb" (UID: "add60ecd-ac12-4d36-88f1-c687bdf4aaeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:33:55.404317 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.404285 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "add60ecd-ac12-4d36-88f1-c687bdf4aaeb" (UID: "add60ecd-ac12-4d36-88f1-c687bdf4aaeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 14:33:55.404317 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.404308 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "add60ecd-ac12-4d36-88f1-c687bdf4aaeb" (UID: "add60ecd-ac12-4d36-88f1-c687bdf4aaeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:33:55.404456 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.404339 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-kube-api-access-5nsnd" (OuterVolumeSpecName: "kube-api-access-5nsnd") pod "add60ecd-ac12-4d36-88f1-c687bdf4aaeb" (UID: "add60ecd-ac12-4d36-88f1-c687bdf4aaeb"). InnerVolumeSpecName "kube-api-access-5nsnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:33:55.404959 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.404943 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "add60ecd-ac12-4d36-88f1-c687bdf4aaeb" (UID: "add60ecd-ac12-4d36-88f1-c687bdf4aaeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:33:55.503623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.503585 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-envoy\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:55.503623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.503620 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nsnd\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-kube-api-access-5nsnd\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:55.503857 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.503635 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-podinfo\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:55.503857 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.503647 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-data\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:55.503857 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.503660 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-credential-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:55.503857 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.503681 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add60ecd-ac12-4d36-88f1-c687bdf4aaeb-istio-token\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:33:55.992806 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.992771 2560 generic.go:356] "Generic (PLEG): container finished" podID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerID="f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea" exitCode=0 Jul 29 14:33:55.992988 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.992817 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" Jul 29 14:33:55.992988 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.992848 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" event={"ID":"add60ecd-ac12-4d36-88f1-c687bdf4aaeb","Type":"ContainerDied","Data":"f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea"} Jul 29 14:33:55.992988 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.992907 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8" event={"ID":"add60ecd-ac12-4d36-88f1-c687bdf4aaeb","Type":"ContainerDied","Data":"1ad426cebc75abc08b2c8061e5682a47605cc50dfcf455f02a73f46e3345ab35"} Jul 29 14:33:55.992988 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:55.992933 2560 scope.go:117] "RemoveContainer" containerID="f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea" Jul 29 14:33:56.003997 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:56.003975 2560 scope.go:117] "RemoveContainer" containerID="d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3" Jul 29 14:33:56.013641 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:56.013620 2560 scope.go:117] "RemoveContainer" containerID="f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea" Jul 29 14:33:56.014065 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:33:56.014034 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea\": container with ID starting with f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea not found: ID does not exist" containerID="f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea" Jul 29 14:33:56.014198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:56.014075 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea"} err="failed to get container status \"f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea\": rpc error: code = NotFound desc = could not find container \"f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea\": container with ID starting with f94840996630cdc35dc0c8b0ce852dcf04e3c7cc6818191151dc4d06852e26ea not found: ID does not exist" Jul 29 14:33:56.014198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:56.014101 2560 scope.go:117] "RemoveContainer" containerID="d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3" Jul 29 14:33:56.014502 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:33:56.014431 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3\": container with ID starting with d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3 not found: ID does not exist" containerID="d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3" Jul 29 14:33:56.014502 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:56.014467 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3"} err="failed to get container status \"d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3\": rpc error: code = NotFound desc = could not find container \"d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3\": container with ID starting with d256845b232f37f8b47e012b2302219fcde8bcdd436506df3060682aa324a3c3 not found: ID does not exist" Jul 29 14:33:56.017514 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:56.017490 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8"] Jul 29 14:33:56.020209 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:56.020187 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-10f19eac-openshift-default-54f568bcd8-gd2p8"] Jul 29 14:33:57.381395 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:33:57.381363 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" path="/var/lib/kubelet/pods/add60ecd-ac12-4d36-88f1-c687bdf4aaeb/volumes" Jul 29 14:34:12.986712 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.986672 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7"] Jul 29 14:34:12.987222 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.987157 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" Jul 29 14:34:12.987222 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.987171 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" Jul 29 14:34:12.987222 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.987182 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" Jul 29 14:34:12.987222 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.987187 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" Jul 29 14:34:12.987222 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.987201 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e36740-87ac-4f5b-ab54-9a8e11add411" containerName="revoke-keys" Jul 29 14:34:12.987222 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.987207 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e36740-87ac-4f5b-ab54-9a8e11add411" containerName="revoke-keys" Jul 29 14:34:12.987611 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.987289 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" Jul 29 14:34:12.987611 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.987299 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e36740-87ac-4f5b-ab54-9a8e11add411" containerName="revoke-keys" Jul 29 14:34:12.987611 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.987308 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="add60ecd-ac12-4d36-88f1-c687bdf4aaeb" containerName="istio-proxy" Jul 29 14:34:12.989528 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.989497 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:12.992109 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.992077 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c3fb6290-serving-cert\"" Jul 29 14:34:12.992336 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:12.992226 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jqfw\"" Jul 29 14:34:13.002407 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.002381 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7"] Jul 29 14:34:13.176253 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.176212 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdg9\" (UniqueName: \"kubernetes.io/projected/67d9a6e8-1abb-4452-a774-ed91732b9541-kube-api-access-vbdg9\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.176473 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.176274 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d9a6e8-1abb-4452-a774-ed91732b9541-maas-api-tls\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.176473 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.176313 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9a6e8-1abb-4452-a774-ed91732b9541-openshift-service-ca\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.277749 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.277655 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d9a6e8-1abb-4452-a774-ed91732b9541-maas-api-tls\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.277749 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.277711 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9a6e8-1abb-4452-a774-ed91732b9541-openshift-service-ca\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.277958 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.277792 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdg9\" (UniqueName: \"kubernetes.io/projected/67d9a6e8-1abb-4452-a774-ed91732b9541-kube-api-access-vbdg9\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.277958 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:34:13.277816 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-serving-cert: secret "maas-api-e2e-mt-c3fb6290-serving-cert" not found Jul 29 14:34:13.277958 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:34:13.277891 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d9a6e8-1abb-4452-a774-ed91732b9541-maas-api-tls podName:67d9a6e8-1abb-4452-a774-ed91732b9541 nodeName:}" failed. No retries permitted until 2026-07-29 14:34:13.777867694 +0000 UTC m=+1986.979368847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67d9a6e8-1abb-4452-a774-ed91732b9541-maas-api-tls") pod "maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" (UID: "67d9a6e8-1abb-4452-a774-ed91732b9541") : secret "maas-api-e2e-mt-c3fb6290-serving-cert" not found Jul 29 14:34:13.278499 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.278481 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9a6e8-1abb-4452-a774-ed91732b9541-openshift-service-ca\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.286796 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.286770 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdg9\" (UniqueName: \"kubernetes.io/projected/67d9a6e8-1abb-4452-a774-ed91732b9541-kube-api-access-vbdg9\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.782706 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.782670 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d9a6e8-1abb-4452-a774-ed91732b9541-maas-api-tls\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.785612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.785584 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d9a6e8-1abb-4452-a774-ed91732b9541-maas-api-tls\") pod \"maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:13.903494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:13.903455 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:14.051868 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:14.051835 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7"] Jul 29 14:34:14.054381 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:34:14.054358 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d9a6e8_1abb_4452_a774_ed91732b9541.slice/crio-f0a2f74a71b7a62fb64a42f555e81030d617968d7040f68567f783f53872dde7 WatchSource:0}: Error finding container f0a2f74a71b7a62fb64a42f555e81030d617968d7040f68567f783f53872dde7: Status 404 returned error can't find the container with id f0a2f74a71b7a62fb64a42f555e81030d617968d7040f68567f783f53872dde7 Jul 29 14:34:14.057118 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:14.057097 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:34:14.111553 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:14.111500 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" event={"ID":"67d9a6e8-1abb-4452-a774-ed91732b9541","Type":"ContainerStarted","Data":"f0a2f74a71b7a62fb64a42f555e81030d617968d7040f68567f783f53872dde7"} Jul 29 14:34:17.135633 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:17.135587 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" event={"ID":"67d9a6e8-1abb-4452-a774-ed91732b9541","Type":"ContainerStarted","Data":"52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677"} Jul 29 14:34:17.136158 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:17.135774 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:17.154833 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:17.154779 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" podStartSLOduration=2.828208192 podStartE2EDuration="5.154761289s" totalImagesPullingTime="2.326553097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:34:17.151577589 +0000 UTC m=+1990.353078759" watchObservedRunningTime="2026-07-29 14:34:17.154761289 +0000 UTC m=+1990.356262491" Jul 29 14:34:18.448363 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:18.448317 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:34:23.147972 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:23.147941 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:34:59.541606 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:34:59.541561 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:35:28.651738 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:28.651651 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:35:32.803654 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:32.803608 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7"] Jul 29 14:35:32.806512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:32.806487 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" Jul 29 14:35:32.808876 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:32.808852 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:35:32.813610 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:32.813578 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7"] Jul 29 14:35:32.949220 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:32.949174 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c81580d-f5eb-4f62-bdc9-f1651bb023c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7\" (UID: \"6c81580d-f5eb-4f62-bdc9-f1651bb023c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" Jul 29 14:35:33.050016 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:33.049972 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c81580d-f5eb-4f62-bdc9-f1651bb023c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7\" (UID: \"6c81580d-f5eb-4f62-bdc9-f1651bb023c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" Jul 29 14:35:33.050645 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:33.050626 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c81580d-f5eb-4f62-bdc9-f1651bb023c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7\" (UID: \"6c81580d-f5eb-4f62-bdc9-f1651bb023c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" Jul 29 14:35:33.119364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:33.119265 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" Jul 29 14:35:33.263530 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:33.263504 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7"] Jul 29 14:35:33.265157 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:35:33.265129 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c81580d_f5eb_4f62_bdc9_f1651bb023c2.slice/crio-09c7cb83aa6e14832bf52accbde0d1984b41ee42b21e607a071507bbdac2076c WatchSource:0}: Error finding container 09c7cb83aa6e14832bf52accbde0d1984b41ee42b21e607a071507bbdac2076c: Status 404 returned error can't find the container with id 09c7cb83aa6e14832bf52accbde0d1984b41ee42b21e607a071507bbdac2076c Jul 29 14:35:33.632803 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:33.632768 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" event={"ID":"6c81580d-f5eb-4f62-bdc9-f1651bb023c2","Type":"ContainerStarted","Data":"647cdc0e0c3bc94a6d6028af7e3fe9de96c8a8f3b7bc2970d198f0990993fe40"} Jul 29 14:35:33.632979 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:33.632820 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" event={"ID":"6c81580d-f5eb-4f62-bdc9-f1651bb023c2","Type":"ContainerStarted","Data":"09c7cb83aa6e14832bf52accbde0d1984b41ee42b21e607a071507bbdac2076c"} Jul 29 14:35:33.647751 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:33.647705 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" podStartSLOduration=1.647689079 podStartE2EDuration="1.647689079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:33.646197385 +0000 UTC m=+2066.847698570" watchObservedRunningTime="2026-07-29 14:35:33.647689079 +0000 UTC m=+2066.849190250" Jul 29 14:35:34.640678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:34.640639 2560 generic.go:356] "Generic (PLEG): container finished" podID="6c81580d-f5eb-4f62-bdc9-f1651bb023c2" containerID="647cdc0e0c3bc94a6d6028af7e3fe9de96c8a8f3b7bc2970d198f0990993fe40" exitCode=0 Jul 29 14:35:34.641078 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:34.640735 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" event={"ID":"6c81580d-f5eb-4f62-bdc9-f1651bb023c2","Type":"ContainerDied","Data":"647cdc0e0c3bc94a6d6028af7e3fe9de96c8a8f3b7bc2970d198f0990993fe40"} Jul 29 14:35:35.793291 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:35.793264 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" Jul 29 14:35:35.876470 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:35.876428 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c81580d-f5eb-4f62-bdc9-f1651bb023c2-maas-api-service-ca\") pod \"6c81580d-f5eb-4f62-bdc9-f1651bb023c2\" (UID: \"6c81580d-f5eb-4f62-bdc9-f1651bb023c2\") " Jul 29 14:35:35.876894 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:35.876865 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c81580d-f5eb-4f62-bdc9-f1651bb023c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c81580d-f5eb-4f62-bdc9-f1651bb023c2" (UID: "6c81580d-f5eb-4f62-bdc9-f1651bb023c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:35:35.977960 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:35.977922 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c81580d-f5eb-4f62-bdc9-f1651bb023c2-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:35:36.654181 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:36.654154 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" Jul 29 14:35:36.654368 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:36.654198 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7" event={"ID":"6c81580d-f5eb-4f62-bdc9-f1651bb023c2","Type":"ContainerDied","Data":"09c7cb83aa6e14832bf52accbde0d1984b41ee42b21e607a071507bbdac2076c"} Jul 29 14:35:36.654368 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:36.654251 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09c7cb83aa6e14832bf52accbde0d1984b41ee42b21e607a071507bbdac2076c" Jul 29 14:35:46.847890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:46.847848 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:35:48.370365 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.370315 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7"] Jul 29 14:35:48.370882 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.370619 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" podUID="67d9a6e8-1abb-4452-a774-ed91732b9541" containerName="maas-api" containerID="cri-o://52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677" gracePeriod=30 Jul 29 14:35:48.599823 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.599407 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7"] Jul 29 14:35:48.602201 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.602172 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3fb6290-s5kz7"] Jul 29 14:35:48.650227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.650199 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:35:48.700462 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.700414 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d9a6e8-1abb-4452-a774-ed91732b9541-maas-api-tls\") pod \"67d9a6e8-1abb-4452-a774-ed91732b9541\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " Jul 29 14:35:48.700680 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.700524 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbdg9\" (UniqueName: \"kubernetes.io/projected/67d9a6e8-1abb-4452-a774-ed91732b9541-kube-api-access-vbdg9\") pod \"67d9a6e8-1abb-4452-a774-ed91732b9541\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " Jul 29 14:35:48.700680 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.700615 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9a6e8-1abb-4452-a774-ed91732b9541-openshift-service-ca\") pod \"67d9a6e8-1abb-4452-a774-ed91732b9541\" (UID: \"67d9a6e8-1abb-4452-a774-ed91732b9541\") " Jul 29 14:35:48.700981 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.700950 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d9a6e8-1abb-4452-a774-ed91732b9541-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67d9a6e8-1abb-4452-a774-ed91732b9541" (UID: "67d9a6e8-1abb-4452-a774-ed91732b9541"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:35:48.703065 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.703033 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d9a6e8-1abb-4452-a774-ed91732b9541-kube-api-access-vbdg9" (OuterVolumeSpecName: "kube-api-access-vbdg9") pod "67d9a6e8-1abb-4452-a774-ed91732b9541" (UID: "67d9a6e8-1abb-4452-a774-ed91732b9541"). InnerVolumeSpecName "kube-api-access-vbdg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:35:48.703165 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.703147 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d9a6e8-1abb-4452-a774-ed91732b9541-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67d9a6e8-1abb-4452-a774-ed91732b9541" (UID: "67d9a6e8-1abb-4452-a774-ed91732b9541"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:35:48.736895 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.736865 2560 generic.go:356] "Generic (PLEG): container finished" podID="67d9a6e8-1abb-4452-a774-ed91732b9541" containerID="52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677" exitCode=0 Jul 29 14:35:48.737050 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.736910 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" Jul 29 14:35:48.737050 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.736956 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" event={"ID":"67d9a6e8-1abb-4452-a774-ed91732b9541","Type":"ContainerDied","Data":"52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677"} Jul 29 14:35:48.737050 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.737007 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7" event={"ID":"67d9a6e8-1abb-4452-a774-ed91732b9541","Type":"ContainerDied","Data":"f0a2f74a71b7a62fb64a42f555e81030d617968d7040f68567f783f53872dde7"} Jul 29 14:35:48.737050 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.737037 2560 scope.go:117] "RemoveContainer" containerID="52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677" Jul 29 14:35:48.748237 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.748212 2560 scope.go:117] "RemoveContainer" containerID="52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677" Jul 29 14:35:48.748666 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:35:48.748644 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677\": container with ID starting with 52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677 not found: ID does not exist" containerID="52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677" Jul 29 14:35:48.748775 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.748674 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677"} err="failed to get container status \"52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677\": rpc error: code = NotFound desc = could not find container \"52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677\": container with ID starting with 52122c3d627ae30127ec804de253233433a1130f08bd9e6ba41dbd1ff15d3677 not found: ID does not exist" Jul 29 14:35:48.761811 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.761779 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7"] Jul 29 14:35:48.765911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.765880 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3fb6290-bbc8d4984-k8kh7"] Jul 29 14:35:48.801800 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.801756 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbdg9\" (UniqueName: \"kubernetes.io/projected/67d9a6e8-1abb-4452-a774-ed91732b9541-kube-api-access-vbdg9\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:35:48.801800 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.801799 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9a6e8-1abb-4452-a774-ed91732b9541-openshift-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:35:48.802014 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:48.801816 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d9a6e8-1abb-4452-a774-ed91732b9541-maas-api-tls\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:35:49.382656 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:49.382621 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d9a6e8-1abb-4452-a774-ed91732b9541" path="/var/lib/kubelet/pods/67d9a6e8-1abb-4452-a774-ed91732b9541/volumes" Jul 29 14:35:49.383052 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:49.382970 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c81580d-f5eb-4f62-bdc9-f1651bb023c2" path="/var/lib/kubelet/pods/6c81580d-f5eb-4f62-bdc9-f1651bb023c2/volumes" Jul 29 14:35:56.044859 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:35:56.044814 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:36:07.557339 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:36:07.557295 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:36:07.560579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:36:07.560530 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:36:07.574314 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:36:07.574285 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:36:07.578310 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:36:07.578280 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:36:07.641895 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:36:07.641799 2560 scope.go:117] "RemoveContainer" containerID="2c37ce88f0586b842042d7b3ddc6e14970c7122c01117e69f50e80f5b60ce842" Jul 29 14:36:07.652002 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:36:07.651980 2560 scope.go:117] "RemoveContainer" containerID="c48d1edb7c84fde69b5e2a2a8c75529733274d9002a8b3dc3beb373c16261a52" Jul 29 14:36:07.664028 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:36:07.664008 2560 scope.go:117] "RemoveContainer" containerID="1c6c2ad3ed96b9b898084a176e15d97363bcf8313f9ad7f1951c00eacb88bb32" Jul 29 14:36:43.946820 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:36:43.946773 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:36:55.057451 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:36:55.057401 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:37:07.700220 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:07.700185 2560 scope.go:117] "RemoveContainer" containerID="c915b3236e3a61fde75cc5260b5b62999ba26158ad8276fbcbfe68239ed6cbaf" Jul 29 14:37:36.385879 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.385841 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg"] Jul 29 14:37:36.386521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.386317 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d9a6e8-1abb-4452-a774-ed91732b9541" containerName="maas-api" Jul 29 14:37:36.386521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.386330 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d9a6e8-1abb-4452-a774-ed91732b9541" containerName="maas-api" Jul 29 14:37:36.386521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.386358 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c81580d-f5eb-4f62-bdc9-f1651bb023c2" containerName="revoke-keys" Jul 29 14:37:36.386521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.386368 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c81580d-f5eb-4f62-bdc9-f1651bb023c2" containerName="revoke-keys" Jul 29 14:37:36.386521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.386445 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d9a6e8-1abb-4452-a774-ed91732b9541" containerName="maas-api" Jul 29 14:37:36.386521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.386458 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c81580d-f5eb-4f62-bdc9-f1651bb023c2" containerName="revoke-keys" Jul 29 14:37:36.388958 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.388930 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.391945 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.391917 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-055496-openshift-default-dockercfg-6g74j\"" Jul 29 14:37:36.402332 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.402303 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg"] Jul 29 14:37:36.545010 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.544970 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-token\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.545189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.545024 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-podinfo\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.545189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.545050 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-socket\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.545189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.545098 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxw4b\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-kube-api-access-bxw4b\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.545189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.545118 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-data\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.545189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.545143 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istiod-ca-cert\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.545189 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.545165 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-envoy\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.545420 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.545266 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-certs\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.545420 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.545318 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-credential-socket\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.646359 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646272 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-token\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.646359 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646314 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-podinfo\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.646359 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646335 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-socket\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.646689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646376 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxw4b\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-kube-api-access-bxw4b\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.646689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646404 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-data\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.646689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646440 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istiod-ca-cert\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.646689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646474 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-envoy\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.646689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646509 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-certs\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.646689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646604 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-credential-socket\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.647011 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646868 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-socket\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.647011 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.646924 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-data\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.647128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.647082 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-credential-socket\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.647286 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.647263 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-certs\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.647584 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.647531 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istiod-ca-cert\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.649236 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.649210 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-podinfo\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.649364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.649346 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-envoy\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.655257 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.655229 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-token\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.655521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.655496 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxw4b\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-kube-api-access-bxw4b\") pod \"e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.703702 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.703657 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:36.872805 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.872776 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg"] Jul 29 14:37:36.875228 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:37:36.875178 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9d5ee1_9e63_4961_a39a_1a4fd7d06c7c.slice/crio-82791668edc5fb33ed4c1a995d8abdaa1eaebb56d89a2d56179503d44beef4c6 WatchSource:0}: Error finding container 82791668edc5fb33ed4c1a995d8abdaa1eaebb56d89a2d56179503d44beef4c6: Status 404 returned error can't find the container with id 82791668edc5fb33ed4c1a995d8abdaa1eaebb56d89a2d56179503d44beef4c6 Jul 29 14:37:36.877868 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.877825 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:37:36.877968 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.877933 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:37:36.878012 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:36.877979 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:37:37.422389 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:37.422345 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" event={"ID":"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c","Type":"ContainerStarted","Data":"d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d"} Jul 29 14:37:37.422849 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:37.422410 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" event={"ID":"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c","Type":"ContainerStarted","Data":"82791668edc5fb33ed4c1a995d8abdaa1eaebb56d89a2d56179503d44beef4c6"} Jul 29 14:37:37.476502 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:37.476438 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" podStartSLOduration=1.476420578 podStartE2EDuration="1.476420578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:37:37.470945912 +0000 UTC m=+2190.672447084" watchObservedRunningTime="2026-07-29 14:37:37.476420578 +0000 UTC m=+2190.677921750" Jul 29 14:37:37.704792 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:37.704753 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:37.709968 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:37.709940 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:38.428504 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:38.428465 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:38.429518 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:38.429497 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:46.747212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:46.747169 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:37:54.470785 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:54.470733 2560 patch_prober.go:28] interesting pod/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 14:37:54.472091 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:54.470828 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 14:37:54.538583 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:54.538521 2560 generic.go:356] "Generic (PLEG): container finished" podID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerID="d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d" exitCode=0 Jul 29 14:37:54.538583 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:54.538571 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" event={"ID":"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c","Type":"ContainerDied","Data":"d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d"} Jul 29 14:37:54.539081 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:54.539042 2560 scope.go:117] "RemoveContainer" containerID="d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d" Jul 29 14:37:54.540221 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:54.540193 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:37:54.540301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:54.540284 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:37:54.540340 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:54.540328 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:37:55.547330 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:55.547298 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" event={"ID":"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c","Type":"ContainerStarted","Data":"5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa"} Jul 29 14:37:55.704701 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:55.704653 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:55.704972 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:55.704942 2560 patch_prober.go:28] interesting pod/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 29 14:37:55.705073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:55.705009 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 29 14:37:56.704649 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:56.704602 2560 patch_prober.go:28] interesting pod/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 29 14:37:56.705195 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:56.704670 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 29 14:37:57.704432 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:57.704383 2560 patch_prober.go:28] interesting pod/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 29 14:37:57.704688 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:57.704457 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 29 14:37:58.704415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:58.704365 2560 patch_prober.go:28] interesting pod/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 29 14:37:58.704652 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:58.704436 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 29 14:37:59.707719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:59.707685 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:59.708132 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:59.708085 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:37:59.708830 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:37:59.708808 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:38:07.727162 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:07.727128 2560 scope.go:117] "RemoveContainer" containerID="7a26fdbc858a4be30c56c261f83cba1a4723b44bdd50360b36b74d246f026156" Jul 29 14:38:08.014076 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.013986 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk"] Jul 29 14:38:08.023201 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.023177 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.025625 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.025599 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-02f676-serving-cert\"" Jul 29 14:38:08.025788 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.025604 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jqfw\"" Jul 29 14:38:08.025962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.025944 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk"] Jul 29 14:38:08.049608 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.049569 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/cc2260a4-8767-4e87-86a2-27c2c2256c7b-kube-api-access-5w9ng\") pod \"maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.049810 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.049635 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2260a4-8767-4e87-86a2-27c2c2256c7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.049810 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.049675 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2260a4-8767-4e87-86a2-27c2c2256c7b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.150753 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.150704 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/cc2260a4-8767-4e87-86a2-27c2c2256c7b-kube-api-access-5w9ng\") pod \"maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.150970 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.150762 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2260a4-8767-4e87-86a2-27c2c2256c7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.150970 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.150800 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2260a4-8767-4e87-86a2-27c2c2256c7b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.151515 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.151491 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2260a4-8767-4e87-86a2-27c2c2256c7b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.153983 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.153957 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2260a4-8767-4e87-86a2-27c2c2256c7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.164671 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.164636 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/cc2260a4-8767-4e87-86a2-27c2c2256c7b-kube-api-access-5w9ng\") pod \"maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.335751 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.335648 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:08.485937 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.485910 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk"] Jul 29 14:38:08.487909 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:38:08.487861 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2260a4_8767_4e87_86a2_27c2c2256c7b.slice/crio-c21dbd90ff3975862f924896abf725e131940c7f61b5c52d8501e9bb4440ad52 WatchSource:0}: Error finding container c21dbd90ff3975862f924896abf725e131940c7f61b5c52d8501e9bb4440ad52: Status 404 returned error can't find the container with id c21dbd90ff3975862f924896abf725e131940c7f61b5c52d8501e9bb4440ad52 Jul 29 14:38:08.628515 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:08.628424 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" event={"ID":"cc2260a4-8767-4e87-86a2-27c2c2256c7b","Type":"ContainerStarted","Data":"c21dbd90ff3975862f924896abf725e131940c7f61b5c52d8501e9bb4440ad52"} Jul 29 14:38:09.636702 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:09.636663 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" event={"ID":"cc2260a4-8767-4e87-86a2-27c2c2256c7b","Type":"ContainerStarted","Data":"6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d"} Jul 29 14:38:09.637094 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:09.636769 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:09.694526 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:09.694464 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" podStartSLOduration=2.383498808 podStartE2EDuration="2.694446701s" totalImagesPullingTime="310.947893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:38:09.690447109 +0000 UTC m=+2222.891948280" watchObservedRunningTime="2026-07-29 14:38:09.694446701 +0000 UTC m=+2222.895947945" Jul 29 14:38:14.954932 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:14.954895 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:38:15.649218 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:15.649187 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:38:54.748896 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:38:54.748813 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:39:06.344556 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:06.344504 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:39:25.453444 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:25.453404 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:39:58.926753 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:58.926669 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4"] Jul 29 14:39:58.930563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:58.930525 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" Jul 29 14:39:58.933092 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:58.933070 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:39:58.938311 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:58.938263 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4"] Jul 29 14:39:59.047212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:59.047169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5857ba6-329d-4368-af31-46441f713673-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-055496-tjtd4\" (UID: \"b5857ba6-329d-4368-af31-46441f713673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" Jul 29 14:39:59.148356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:59.148316 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5857ba6-329d-4368-af31-46441f713673-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-055496-tjtd4\" (UID: \"b5857ba6-329d-4368-af31-46441f713673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" Jul 29 14:39:59.149057 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:59.149029 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5857ba6-329d-4368-af31-46441f713673-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-055496-tjtd4\" (UID: \"b5857ba6-329d-4368-af31-46441f713673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" Jul 29 14:39:59.243761 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:59.243663 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" Jul 29 14:39:59.395802 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:39:59.395773 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4"] Jul 29 14:39:59.398136 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:39:59.398106 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5857ba6_329d_4368_af31_46441f713673.slice/crio-449de4a37b5102046026f55976667efb1705a27eae090e38d76b65ac6aa36bc1 WatchSource:0}: Error finding container 449de4a37b5102046026f55976667efb1705a27eae090e38d76b65ac6aa36bc1: Status 404 returned error can't find the container with id 449de4a37b5102046026f55976667efb1705a27eae090e38d76b65ac6aa36bc1 Jul 29 14:40:00.330574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:00.330513 2560 generic.go:356] "Generic (PLEG): container finished" podID="b5857ba6-329d-4368-af31-46441f713673" containerID="68873419f4fa5e1d753c8354f27ae5d89cbc9990fe8e3c6d50473ba4bb0f4574" exitCode=0 Jul 29 14:40:00.331030 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:00.330605 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" event={"ID":"b5857ba6-329d-4368-af31-46441f713673","Type":"ContainerDied","Data":"68873419f4fa5e1d753c8354f27ae5d89cbc9990fe8e3c6d50473ba4bb0f4574"} Jul 29 14:40:00.331030 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:00.330656 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" event={"ID":"b5857ba6-329d-4368-af31-46441f713673","Type":"ContainerStarted","Data":"449de4a37b5102046026f55976667efb1705a27eae090e38d76b65ac6aa36bc1"} Jul 29 14:40:01.481779 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:01.481752 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" Jul 29 14:40:01.570824 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:01.570781 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5857ba6-329d-4368-af31-46441f713673-maas-api-service-ca\") pod \"b5857ba6-329d-4368-af31-46441f713673\" (UID: \"b5857ba6-329d-4368-af31-46441f713673\") " Jul 29 14:40:01.571141 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:01.571117 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5857ba6-329d-4368-af31-46441f713673-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5857ba6-329d-4368-af31-46441f713673" (UID: "b5857ba6-329d-4368-af31-46441f713673"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:40:01.672227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:01.672140 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5857ba6-329d-4368-af31-46441f713673-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:02.343962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:02.343927 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" Jul 29 14:40:02.344159 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:02.343960 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4" event={"ID":"b5857ba6-329d-4368-af31-46441f713673","Type":"ContainerDied","Data":"449de4a37b5102046026f55976667efb1705a27eae090e38d76b65ac6aa36bc1"} Jul 29 14:40:02.344159 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:02.344013 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="449de4a37b5102046026f55976667efb1705a27eae090e38d76b65ac6aa36bc1" Jul 29 14:40:03.245271 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:03.245231 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:40:07.759831 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:07.759799 2560 scope.go:117] "RemoveContainer" containerID="ba9b3cee6885c2a9e886a29e466b966a095af2007ad6914e1758ee6ab5746083" Jul 29 14:40:09.787624 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:09.787582 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4"] Jul 29 14:40:09.790734 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:09.790704 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-055496-tjtd4"] Jul 29 14:40:11.381224 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:11.381187 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5857ba6-329d-4368-af31-46441f713673" path="/var/lib/kubelet/pods/b5857ba6-329d-4368-af31-46441f713673/volumes" Jul 29 14:40:16.198552 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.198499 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg"] Jul 29 14:40:16.198967 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.198786 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" containerID="cri-o://5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa" gracePeriod=30 Jul 29 14:40:16.567209 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.567117 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts"] Jul 29 14:40:16.567591 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.567579 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5857ba6-329d-4368-af31-46441f713673" containerName="revoke-keys" Jul 29 14:40:16.567641 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.567594 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5857ba6-329d-4368-af31-46441f713673" containerName="revoke-keys" Jul 29 14:40:16.567684 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.567680 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5857ba6-329d-4368-af31-46441f713673" containerName="revoke-keys" Jul 29 14:40:16.570939 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.570921 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" Jul 29 14:40:16.573487 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.573460 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:40:16.578082 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.578051 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts"] Jul 29 14:40:16.612163 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.612125 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924da866-45f9-4a64-85cc-2b8a5c73a259-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts\" (UID: \"924da866-45f9-4a64-85cc-2b8a5c73a259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" Jul 29 14:40:16.713289 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.713236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924da866-45f9-4a64-85cc-2b8a5c73a259-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts\" (UID: \"924da866-45f9-4a64-85cc-2b8a5c73a259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" Jul 29 14:40:16.713929 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.713908 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924da866-45f9-4a64-85cc-2b8a5c73a259-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts\" (UID: \"924da866-45f9-4a64-85cc-2b8a5c73a259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" Jul 29 14:40:16.884496 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:16.884369 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" Jul 29 14:40:17.059467 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:17.059431 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts"] Jul 29 14:40:17.061091 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:40:17.061059 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924da866_45f9_4a64_85cc_2b8a5c73a259.slice/crio-82769161ea38b4f846a1d54515ac99730d28ac32db0bfa380863cb05af5c7cdb WatchSource:0}: Error finding container 82769161ea38b4f846a1d54515ac99730d28ac32db0bfa380863cb05af5c7cdb: Status 404 returned error can't find the container with id 82769161ea38b4f846a1d54515ac99730d28ac32db0bfa380863cb05af5c7cdb Jul 29 14:40:17.441461 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:17.441420 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" event={"ID":"924da866-45f9-4a64-85cc-2b8a5c73a259","Type":"ContainerStarted","Data":"4c22a58c3f3143bfbeb2a080ba8c035367dd4a451927da624a58a4bd541fc301"} Jul 29 14:40:17.441871 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:17.441486 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" event={"ID":"924da866-45f9-4a64-85cc-2b8a5c73a259","Type":"ContainerStarted","Data":"82769161ea38b4f846a1d54515ac99730d28ac32db0bfa380863cb05af5c7cdb"} Jul 29 14:40:17.458690 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:17.458635 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" podStartSLOduration=1.458617958 podStartE2EDuration="1.458617958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:40:17.456450934 +0000 UTC m=+2350.657952109" watchObservedRunningTime="2026-07-29 14:40:17.458617958 +0000 UTC m=+2350.660119130" Jul 29 14:40:18.449305 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:18.449266 2560 generic.go:356] "Generic (PLEG): container finished" podID="924da866-45f9-4a64-85cc-2b8a5c73a259" containerID="4c22a58c3f3143bfbeb2a080ba8c035367dd4a451927da624a58a4bd541fc301" exitCode=0 Jul 29 14:40:18.449769 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:18.449313 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" event={"ID":"924da866-45f9-4a64-85cc-2b8a5c73a259","Type":"ContainerDied","Data":"4c22a58c3f3143bfbeb2a080ba8c035367dd4a451927da624a58a4bd541fc301"} Jul 29 14:40:19.461341 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:19.461307 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:40:19.605196 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:19.605155 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" Jul 29 14:40:19.644381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:19.644345 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924da866-45f9-4a64-85cc-2b8a5c73a259-maas-api-service-ca\") pod \"924da866-45f9-4a64-85cc-2b8a5c73a259\" (UID: \"924da866-45f9-4a64-85cc-2b8a5c73a259\") " Jul 29 14:40:19.644747 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:19.644727 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924da866-45f9-4a64-85cc-2b8a5c73a259-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "924da866-45f9-4a64-85cc-2b8a5c73a259" (UID: "924da866-45f9-4a64-85cc-2b8a5c73a259"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:40:19.746199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:19.746101 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924da866-45f9-4a64-85cc-2b8a5c73a259-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:20.463762 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:20.463725 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" event={"ID":"924da866-45f9-4a64-85cc-2b8a5c73a259","Type":"ContainerDied","Data":"82769161ea38b4f846a1d54515ac99730d28ac32db0bfa380863cb05af5c7cdb"} Jul 29 14:40:20.463762 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:20.463765 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts" Jul 29 14:40:20.464338 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:20.463775 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82769161ea38b4f846a1d54515ac99730d28ac32db0bfa380863cb05af5c7cdb" Jul 29 14:40:21.468207 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.468140 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:40:21.471242 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.471214 2560 generic.go:356] "Generic (PLEG): container finished" podID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerID="5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa" exitCode=0 Jul 29 14:40:21.471384 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.471263 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" Jul 29 14:40:21.471384 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.471300 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" event={"ID":"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c","Type":"ContainerDied","Data":"5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa"} Jul 29 14:40:21.471384 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.471359 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg" event={"ID":"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c","Type":"ContainerDied","Data":"82791668edc5fb33ed4c1a995d8abdaa1eaebb56d89a2d56179503d44beef4c6"} Jul 29 14:40:21.471509 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.471386 2560 scope.go:117] "RemoveContainer" containerID="5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa" Jul 29 14:40:21.481908 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.481881 2560 scope.go:117] "RemoveContainer" containerID="d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d" Jul 29 14:40:21.496204 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.495668 2560 scope.go:117] "RemoveContainer" containerID="5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa" Jul 29 14:40:21.505842 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:40:21.505793 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa\": container with ID starting with 5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa not found: ID does not exist" containerID="5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa" Jul 29 14:40:21.506115 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.506072 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa"} err="failed to get container status \"5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa\": rpc error: code = NotFound desc = could not find container \"5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa\": container with ID starting with 5734b10294f3f054ef72d8242190246ad7602e2a30ad748878219a5bb7d1deaa not found: ID does not exist" Jul 29 14:40:21.506279 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.506263 2560 scope.go:117] "RemoveContainer" containerID="d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d" Jul 29 14:40:21.509300 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:40:21.509240 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d\": container with ID starting with d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d not found: ID does not exist" containerID="d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d" Jul 29 14:40:21.509300 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.509284 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d"} err="failed to get container status \"d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d\": rpc error: code = NotFound desc = could not find container \"d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d\": container with ID starting with d1667f6b4d44c13e36ad7b8922f0b4945009fb4ef135a50c48d28018ce9a8b7d not found: ID does not exist" Jul 29 14:40:21.565599 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565519 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-data\") pod \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " Jul 29 14:40:21.565599 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565611 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-credential-socket\") pod \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " Jul 29 14:40:21.565845 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565666 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-certs\") pod \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " Jul 29 14:40:21.565845 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565719 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-socket\") pod \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " Jul 29 14:40:21.565845 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565781 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-podinfo\") pod \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " Jul 29 14:40:21.565845 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565817 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-envoy\") pod \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " Jul 29 14:40:21.565845 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565846 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istiod-ca-cert\") pod \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " Jul 29 14:40:21.566101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565877 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-token\") pod \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " Jul 29 14:40:21.566101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565879 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" (UID: "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:40:21.566101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.565912 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxw4b\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-kube-api-access-bxw4b\") pod \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\" (UID: \"7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c\") " Jul 29 14:40:21.566101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.566045 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" (UID: "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:40:21.566299 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.566102 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" (UID: "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:40:21.566299 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.566227 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" (UID: "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:40:21.566415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.566322 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:21.566415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.566347 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istiod-ca-cert\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:21.566415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.566364 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-credential-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:21.566415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.566380 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-workload-certs\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:21.566415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.566394 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" (UID: "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:40:21.568234 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.568210 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" (UID: "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:40:21.568353 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.568231 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" (UID: "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 14:40:21.568730 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.568709 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" (UID: "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:40:21.568797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.568738 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-kube-api-access-bxw4b" (OuterVolumeSpecName: "kube-api-access-bxw4b") pod "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" (UID: "7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c"). InnerVolumeSpecName "kube-api-access-bxw4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:40:21.667270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.667234 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxw4b\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-kube-api-access-bxw4b\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:21.667270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.667268 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-data\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:21.667270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.667277 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-podinfo\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:21.667270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.667287 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-envoy\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:21.667603 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.667301 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c-istio-token\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:21.796356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.796324 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg"] Jul 29 14:40:21.800326 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:21.800300 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-055496-openshift-default-5c46cb6cfb-dzhvg"] Jul 29 14:40:23.381148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:23.381111 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" path="/var/lib/kubelet/pods/7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c/volumes" Jul 29 14:40:27.075604 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.072293 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk"] Jul 29 14:40:27.075604 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.072687 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" podUID="cc2260a4-8767-4e87-86a2-27c2c2256c7b" containerName="maas-api" containerID="cri-o://6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d" gracePeriod=30 Jul 29 14:40:27.278530 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.278496 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts"] Jul 29 14:40:27.281322 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.281290 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02f676-hs8ts"] Jul 29 14:40:27.340820 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.340751 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:40:27.381060 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.381027 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="924da866-45f9-4a64-85cc-2b8a5c73a259" path="/var/lib/kubelet/pods/924da866-45f9-4a64-85cc-2b8a5c73a259/volumes" Jul 29 14:40:27.421696 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.421659 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2260a4-8767-4e87-86a2-27c2c2256c7b-openshift-service-ca\") pod \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " Jul 29 14:40:27.421855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.421722 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2260a4-8767-4e87-86a2-27c2c2256c7b-maas-api-tls\") pod \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " Jul 29 14:40:27.421855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.421798 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/cc2260a4-8767-4e87-86a2-27c2c2256c7b-kube-api-access-5w9ng\") pod \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\" (UID: \"cc2260a4-8767-4e87-86a2-27c2c2256c7b\") " Jul 29 14:40:27.422040 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.421996 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2260a4-8767-4e87-86a2-27c2c2256c7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc2260a4-8767-4e87-86a2-27c2c2256c7b" (UID: "cc2260a4-8767-4e87-86a2-27c2c2256c7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:40:27.423977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.423951 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2260a4-8767-4e87-86a2-27c2c2256c7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc2260a4-8767-4e87-86a2-27c2c2256c7b" (UID: "cc2260a4-8767-4e87-86a2-27c2c2256c7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:40:27.424081 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.424037 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2260a4-8767-4e87-86a2-27c2c2256c7b-kube-api-access-5w9ng" (OuterVolumeSpecName: "kube-api-access-5w9ng") pod "cc2260a4-8767-4e87-86a2-27c2c2256c7b" (UID: "cc2260a4-8767-4e87-86a2-27c2c2256c7b"). InnerVolumeSpecName "kube-api-access-5w9ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:40:27.511162 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.511127 2560 generic.go:356] "Generic (PLEG): container finished" podID="cc2260a4-8767-4e87-86a2-27c2c2256c7b" containerID="6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d" exitCode=0 Jul 29 14:40:27.511344 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.511178 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" Jul 29 14:40:27.511344 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.511180 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" event={"ID":"cc2260a4-8767-4e87-86a2-27c2c2256c7b","Type":"ContainerDied","Data":"6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d"} Jul 29 14:40:27.511344 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.511286 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk" event={"ID":"cc2260a4-8767-4e87-86a2-27c2c2256c7b","Type":"ContainerDied","Data":"c21dbd90ff3975862f924896abf725e131940c7f61b5c52d8501e9bb4440ad52"} Jul 29 14:40:27.511344 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.511307 2560 scope.go:117] "RemoveContainer" containerID="6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d" Jul 29 14:40:27.521862 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.521837 2560 scope.go:117] "RemoveContainer" containerID="6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d" Jul 29 14:40:27.522159 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:40:27.522140 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d\": container with ID starting with 6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d not found: ID does not exist" containerID="6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d" Jul 29 14:40:27.522203 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.522169 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d"} err="failed to get container status \"6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d\": rpc error: code = NotFound desc = could not find container \"6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d\": container with ID starting with 6e70dd5208a17fef7f5cf4aa3a74a6041d14f937a2d2c231a9d722f5a07d015d not found: ID does not exist" Jul 29 14:40:27.522452 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.522436 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2260a4-8767-4e87-86a2-27c2c2256c7b-maas-api-tls\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:27.522490 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.522458 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/cc2260a4-8767-4e87-86a2-27c2c2256c7b-kube-api-access-5w9ng\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:27.522490 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.522468 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2260a4-8767-4e87-86a2-27c2c2256c7b-openshift-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:40:27.558835 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.558801 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk"] Jul 29 14:40:27.572341 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:27.572309 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02f676-8565d9bd77-jwvbk"] Jul 29 14:40:29.380695 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:29.380663 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2260a4-8767-4e87-86a2-27c2c2256c7b" path="/var/lib/kubelet/pods/cc2260a4-8767-4e87-86a2-27c2c2256c7b/volumes" Jul 29 14:40:31.243943 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:31.243906 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:40:35.062572 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.062518 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79"] Jul 29 14:40:35.063016 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063000 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2260a4-8767-4e87-86a2-27c2c2256c7b" containerName="maas-api" Jul 29 14:40:35.063061 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063018 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2260a4-8767-4e87-86a2-27c2c2256c7b" containerName="maas-api" Jul 29 14:40:35.063061 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063027 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924da866-45f9-4a64-85cc-2b8a5c73a259" containerName="revoke-keys" Jul 29 14:40:35.063061 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063034 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="924da866-45f9-4a64-85cc-2b8a5c73a259" containerName="revoke-keys" Jul 29 14:40:35.063061 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063051 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" Jul 29 14:40:35.063061 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063057 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" Jul 29 14:40:35.063219 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063125 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2260a4-8767-4e87-86a2-27c2c2256c7b" containerName="maas-api" Jul 29 14:40:35.063219 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063134 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="924da866-45f9-4a64-85cc-2b8a5c73a259" containerName="revoke-keys" Jul 29 14:40:35.063219 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063141 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" Jul 29 14:40:35.063219 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.063148 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" Jul 29 14:40:35.068043 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.068019 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.071516 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.071488 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a20e0b-openshift-default-dockercfg-rzxkc\"" Jul 29 14:40:35.077978 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.077797 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79"] Jul 29 14:40:35.201864 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.201818 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-podinfo\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.201864 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.201859 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-token\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.202144 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.201896 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-socket\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.202144 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.201921 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-envoy\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.202144 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.201972 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-certs\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.202144 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.202011 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-data\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.202310 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.202146 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-credential-socket\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.202310 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.202226 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc16ef99-cece-43a9-b95e-830676ef49d3-istiod-ca-cert\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.202310 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.202286 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m757r\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-kube-api-access-m757r\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.303606 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.303562 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc16ef99-cece-43a9-b95e-830676ef49d3-istiod-ca-cert\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.303814 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.303644 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m757r\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-kube-api-access-m757r\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.303814 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.303686 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-podinfo\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.303814 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.303711 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-token\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.303814 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.303736 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-socket\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.303814 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.303761 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-envoy\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.303814 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.303788 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-certs\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.304129 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.303821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-data\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.304129 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.303932 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-credential-socket\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.304207 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.304148 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-socket\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.304410 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.304387 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc16ef99-cece-43a9-b95e-830676ef49d3-istiod-ca-cert\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.304482 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.304473 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-certs\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.304612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.304583 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-data\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.304612 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.304596 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-credential-socket\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.306315 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.306296 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-envoy\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.306604 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.306584 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-podinfo\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.311827 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.311801 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-token\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.311971 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.311847 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m757r\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-kube-api-access-m757r\") pod \"e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.383212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.383120 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:35.557701 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.557677 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79"] Jul 29 14:40:35.560119 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:40:35.560088 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc16ef99_cece_43a9_b95e_830676ef49d3.slice/crio-da8bd3bd4a75b081e1d5695e90517ae6134212b5ee41a93125f06bcf71f35e19 WatchSource:0}: Error finding container da8bd3bd4a75b081e1d5695e90517ae6134212b5ee41a93125f06bcf71f35e19: Status 404 returned error can't find the container with id da8bd3bd4a75b081e1d5695e90517ae6134212b5ee41a93125f06bcf71f35e19 Jul 29 14:40:35.562418 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.562379 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:40:35.562500 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.562465 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:40:35.562568 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:35.562507 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:40:36.569713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:36.569677 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" event={"ID":"bc16ef99-cece-43a9-b95e-830676ef49d3","Type":"ContainerStarted","Data":"99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21"} Jul 29 14:40:36.570138 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:36.569737 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" event={"ID":"bc16ef99-cece-43a9-b95e-830676ef49d3","Type":"ContainerStarted","Data":"da8bd3bd4a75b081e1d5695e90517ae6134212b5ee41a93125f06bcf71f35e19"} Jul 29 14:40:36.591143 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:36.591078 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" podStartSLOduration=1.591054369 podStartE2EDuration="1.591054369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:40:36.587766526 +0000 UTC m=+2369.789267698" watchObservedRunningTime="2026-07-29 14:40:36.591054369 +0000 UTC m=+2369.792555543" Jul 29 14:40:37.389776 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:37.389744 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:37.389973 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:37.389852 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:37.576007 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:37.575976 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:37.577283 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:37.577258 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:43.049526 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:43.049486 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:40:51.670737 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:51.670700 2560 generic.go:356] "Generic (PLEG): container finished" podID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerID="99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21" exitCode=0 Jul 29 14:40:51.671307 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:51.670778 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" event={"ID":"bc16ef99-cece-43a9-b95e-830676ef49d3","Type":"ContainerDied","Data":"99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21"} Jul 29 14:40:51.671307 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:51.671287 2560 scope.go:117] "RemoveContainer" containerID="99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21" Jul 29 14:40:51.672462 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:51.672438 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:40:51.672532 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:51.672505 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:40:51.672594 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:51.672553 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:40:52.575907 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:52.575867 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:52.678603 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:52.678559 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" event={"ID":"bc16ef99-cece-43a9-b95e-830676ef49d3","Type":"ContainerStarted","Data":"1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8"} Jul 29 14:40:53.383394 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:53.383360 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:53.384803 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:53.384779 2560 patch_prober.go:28] interesting pod/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 29 14:40:53.384919 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:53.384826 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 29 14:40:54.383973 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:54.383938 2560 patch_prober.go:28] interesting pod/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 29 14:40:54.384384 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:54.384010 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 29 14:40:55.384194 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:55.384154 2560 patch_prober.go:28] interesting pod/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 29 14:40:55.384622 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:55.384209 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 29 14:40:56.384527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:56.384487 2560 patch_prober.go:28] interesting pod/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 29 14:40:56.385073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:56.384585 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 29 14:40:57.387699 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:57.387670 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:57.388127 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:57.388014 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:40:57.388782 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:40:57.388762 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:41:07.601983 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:07.601894 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:41:07.605411 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:07.605384 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:41:07.625828 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:07.625802 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:41:07.629485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:07.629462 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:41:15.648791 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:15.648757 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:41:36.167787 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.167629 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg"] Jul 29 14:41:36.168191 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.168148 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" Jul 29 14:41:36.168191 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.168161 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9d5ee1-9e63-4961-a39a-1a4fd7d06c7c" containerName="istio-proxy" Jul 29 14:41:36.171620 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.171599 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" Jul 29 14:41:36.174086 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.174064 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:41:36.178504 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.178475 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg"] Jul 29 14:41:36.207602 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.207529 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfa38e0-998d-4c89-971c-6c932cda9672-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a20e0b-579kg\" (UID: \"abfa38e0-998d-4c89-971c-6c932cda9672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" Jul 29 14:41:36.309041 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.309005 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfa38e0-998d-4c89-971c-6c932cda9672-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a20e0b-579kg\" (UID: \"abfa38e0-998d-4c89-971c-6c932cda9672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" Jul 29 14:41:36.309866 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.309843 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfa38e0-998d-4c89-971c-6c932cda9672-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a20e0b-579kg\" (UID: \"abfa38e0-998d-4c89-971c-6c932cda9672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" Jul 29 14:41:36.483608 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.483568 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" Jul 29 14:41:36.640852 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.640823 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg"] Jul 29 14:41:36.642600 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:41:36.642566 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfa38e0_998d_4c89_971c_6c932cda9672.slice/crio-c240e650d963a170edc94867149acadcf36d9ed4a191c2b029e38e97756aed63 WatchSource:0}: Error finding container c240e650d963a170edc94867149acadcf36d9ed4a191c2b029e38e97756aed63: Status 404 returned error can't find the container with id c240e650d963a170edc94867149acadcf36d9ed4a191c2b029e38e97756aed63 Jul 29 14:41:36.970393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.970358 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" event={"ID":"abfa38e0-998d-4c89-971c-6c932cda9672","Type":"ContainerStarted","Data":"aa1e3bc83f45b0736c680e5e4ec50d8367635218d14532ef75fbb890bc9c23ff"} Jul 29 14:41:36.970614 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.970410 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" event={"ID":"abfa38e0-998d-4c89-971c-6c932cda9672","Type":"ContainerStarted","Data":"c240e650d963a170edc94867149acadcf36d9ed4a191c2b029e38e97756aed63"} Jul 29 14:41:36.985764 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:36.985707 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" podStartSLOduration=0.985687052 podStartE2EDuration="985.687052ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:41:36.984692234 +0000 UTC m=+2430.186193407" watchObservedRunningTime="2026-07-29 14:41:36.985687052 +0000 UTC m=+2430.187188224" Jul 29 14:41:37.978699 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:37.978667 2560 generic.go:356] "Generic (PLEG): container finished" podID="abfa38e0-998d-4c89-971c-6c932cda9672" containerID="aa1e3bc83f45b0736c680e5e4ec50d8367635218d14532ef75fbb890bc9c23ff" exitCode=0 Jul 29 14:41:37.979232 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:37.978757 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" event={"ID":"abfa38e0-998d-4c89-971c-6c932cda9672","Type":"ContainerDied","Data":"aa1e3bc83f45b0736c680e5e4ec50d8367635218d14532ef75fbb890bc9c23ff"} Jul 29 14:41:39.133021 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:39.132992 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" Jul 29 14:41:39.238789 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:39.238697 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfa38e0-998d-4c89-971c-6c932cda9672-maas-api-service-ca\") pod \"abfa38e0-998d-4c89-971c-6c932cda9672\" (UID: \"abfa38e0-998d-4c89-971c-6c932cda9672\") " Jul 29 14:41:39.239028 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:39.239005 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfa38e0-998d-4c89-971c-6c932cda9672-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abfa38e0-998d-4c89-971c-6c932cda9672" (UID: "abfa38e0-998d-4c89-971c-6c932cda9672"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:41:39.340450 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:39.340409 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfa38e0-998d-4c89-971c-6c932cda9672-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:41:39.993188 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:39.993156 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" Jul 29 14:41:39.993188 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:39.993170 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg" event={"ID":"abfa38e0-998d-4c89-971c-6c932cda9672","Type":"ContainerDied","Data":"c240e650d963a170edc94867149acadcf36d9ed4a191c2b029e38e97756aed63"} Jul 29 14:41:39.993413 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:39.993214 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c240e650d963a170edc94867149acadcf36d9ed4a191c2b029e38e97756aed63" Jul 29 14:41:51.918159 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:51.918119 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg"] Jul 29 14:41:51.922440 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:51.922405 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a20e0b-579kg"] Jul 29 14:41:53.380768 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:53.380731 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfa38e0-998d-4c89-971c-6c932cda9672" path="/var/lib/kubelet/pods/abfa38e0-998d-4c89-971c-6c932cda9672/volumes" Jul 29 14:41:54.770694 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:54.770655 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:41:58.142673 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.142638 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79"] Jul 29 14:41:58.143151 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.142864 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" containerID="cri-o://1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8" gracePeriod=30 Jul 29 14:41:58.555656 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.555622 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh"] Jul 29 14:41:58.556124 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.556109 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfa38e0-998d-4c89-971c-6c932cda9672" containerName="revoke-keys" Jul 29 14:41:58.556182 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.556126 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfa38e0-998d-4c89-971c-6c932cda9672" containerName="revoke-keys" Jul 29 14:41:58.556216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.556211 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfa38e0-998d-4c89-971c-6c932cda9672" containerName="revoke-keys" Jul 29 14:41:58.559525 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.559507 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" Jul 29 14:41:58.562341 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.562323 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:41:58.572571 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.570611 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh"] Jul 29 14:41:58.623706 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.623664 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4845bcb-6891-418d-b150-92c3c4879dbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ffdf30-575zh\" (UID: \"c4845bcb-6891-418d-b150-92c3c4879dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" Jul 29 14:41:58.724977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.724933 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4845bcb-6891-418d-b150-92c3c4879dbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ffdf30-575zh\" (UID: \"c4845bcb-6891-418d-b150-92c3c4879dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" Jul 29 14:41:58.725688 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.725664 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4845bcb-6891-418d-b150-92c3c4879dbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ffdf30-575zh\" (UID: \"c4845bcb-6891-418d-b150-92c3c4879dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" Jul 29 14:41:58.872206 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:58.872105 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" Jul 29 14:41:59.024096 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:59.024069 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh"] Jul 29 14:41:59.025680 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:41:59.025650 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4845bcb_6891_418d_b150_92c3c4879dbe.slice/crio-e545c9d411058f4b1d8159336ca53c1e24bad914cadc9ad5a64042a2d6e9dbb4 WatchSource:0}: Error finding container e545c9d411058f4b1d8159336ca53c1e24bad914cadc9ad5a64042a2d6e9dbb4: Status 404 returned error can't find the container with id e545c9d411058f4b1d8159336ca53c1e24bad914cadc9ad5a64042a2d6e9dbb4 Jul 29 14:41:59.111301 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:41:59.111268 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" event={"ID":"c4845bcb-6891-418d-b150-92c3c4879dbe","Type":"ContainerStarted","Data":"e545c9d411058f4b1d8159336ca53c1e24bad914cadc9ad5a64042a2d6e9dbb4"} Jul 29 14:42:00.118848 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:00.118753 2560 generic.go:356] "Generic (PLEG): container finished" podID="c4845bcb-6891-418d-b150-92c3c4879dbe" containerID="c598b3ca5885cdad499bf6ed87181d3ce26cff73de7c261697ad183b1355f208" exitCode=0 Jul 29 14:42:00.118848 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:00.118839 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" event={"ID":"c4845bcb-6891-418d-b150-92c3c4879dbe","Type":"ContainerDied","Data":"c598b3ca5885cdad499bf6ed87181d3ce26cff73de7c261697ad183b1355f208"} Jul 29 14:42:01.266443 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:01.266412 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" Jul 29 14:42:01.351914 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:01.351873 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4845bcb-6891-418d-b150-92c3c4879dbe-maas-api-service-ca\") pod \"c4845bcb-6891-418d-b150-92c3c4879dbe\" (UID: \"c4845bcb-6891-418d-b150-92c3c4879dbe\") " Jul 29 14:42:01.352240 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:01.352216 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4845bcb-6891-418d-b150-92c3c4879dbe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4845bcb-6891-418d-b150-92c3c4879dbe" (UID: "c4845bcb-6891-418d-b150-92c3c4879dbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:42:01.453284 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:01.453250 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4845bcb-6891-418d-b150-92c3c4879dbe-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:02.132523 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:02.132491 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" Jul 29 14:42:02.132720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:02.132529 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh" event={"ID":"c4845bcb-6891-418d-b150-92c3c4879dbe","Type":"ContainerDied","Data":"e545c9d411058f4b1d8159336ca53c1e24bad914cadc9ad5a64042a2d6e9dbb4"} Jul 29 14:42:02.132720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:02.132595 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e545c9d411058f4b1d8159336ca53c1e24bad914cadc9ad5a64042a2d6e9dbb4" Jul 29 14:42:03.418813 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.418745 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:42:03.571903 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.571870 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-envoy\") pod \"bc16ef99-cece-43a9-b95e-830676ef49d3\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " Jul 29 14:42:03.572094 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.571917 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-token\") pod \"bc16ef99-cece-43a9-b95e-830676ef49d3\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " Jul 29 14:42:03.572094 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.571944 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-data\") pod \"bc16ef99-cece-43a9-b95e-830676ef49d3\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " Jul 29 14:42:03.572094 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.571966 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc16ef99-cece-43a9-b95e-830676ef49d3-istiod-ca-cert\") pod \"bc16ef99-cece-43a9-b95e-830676ef49d3\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " Jul 29 14:42:03.572094 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572071 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-socket\") pod \"bc16ef99-cece-43a9-b95e-830676ef49d3\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " Jul 29 14:42:03.572319 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572098 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-credential-socket\") pod \"bc16ef99-cece-43a9-b95e-830676ef49d3\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " Jul 29 14:42:03.572319 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572141 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m757r\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-kube-api-access-m757r\") pod \"bc16ef99-cece-43a9-b95e-830676ef49d3\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " Jul 29 14:42:03.572319 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572200 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-certs\") pod \"bc16ef99-cece-43a9-b95e-830676ef49d3\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " Jul 29 14:42:03.572319 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572304 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-podinfo\") pod \"bc16ef99-cece-43a9-b95e-830676ef49d3\" (UID: \"bc16ef99-cece-43a9-b95e-830676ef49d3\") " Jul 29 14:42:03.572527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572404 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc16ef99-cece-43a9-b95e-830676ef49d3" (UID: "bc16ef99-cece-43a9-b95e-830676ef49d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:42:03.572527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572413 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc16ef99-cece-43a9-b95e-830676ef49d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc16ef99-cece-43a9-b95e-830676ef49d3" (UID: "bc16ef99-cece-43a9-b95e-830676ef49d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:42:03.572660 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572640 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc16ef99-cece-43a9-b95e-830676ef49d3" (UID: "bc16ef99-cece-43a9-b95e-830676ef49d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:42:03.572720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572664 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc16ef99-cece-43a9-b95e-830676ef49d3" (UID: "bc16ef99-cece-43a9-b95e-830676ef49d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:42:03.572852 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572832 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc16ef99-cece-43a9-b95e-830676ef49d3-istiod-ca-cert\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:03.572968 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572859 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:03.572968 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572871 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-credential-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:03.572968 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.572880 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-workload-certs\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:03.573297 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.573206 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc16ef99-cece-43a9-b95e-830676ef49d3" (UID: "bc16ef99-cece-43a9-b95e-830676ef49d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:42:03.574415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.574390 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc16ef99-cece-43a9-b95e-830676ef49d3" (UID: "bc16ef99-cece-43a9-b95e-830676ef49d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:42:03.574560 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.574421 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc16ef99-cece-43a9-b95e-830676ef49d3" (UID: "bc16ef99-cece-43a9-b95e-830676ef49d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:42:03.574631 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.574613 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc16ef99-cece-43a9-b95e-830676ef49d3" (UID: "bc16ef99-cece-43a9-b95e-830676ef49d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 14:42:03.574949 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.574932 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-kube-api-access-m757r" (OuterVolumeSpecName: "kube-api-access-m757r") pod "bc16ef99-cece-43a9-b95e-830676ef49d3" (UID: "bc16ef99-cece-43a9-b95e-830676ef49d3"). InnerVolumeSpecName "kube-api-access-m757r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:42:03.674494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.674405 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-podinfo\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:03.674494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.674433 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-envoy\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:03.674494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.674442 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-token\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:03.674494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.674451 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc16ef99-cece-43a9-b95e-830676ef49d3-istio-data\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:03.674494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:03.674460 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m757r\" (UniqueName: \"kubernetes.io/projected/bc16ef99-cece-43a9-b95e-830676ef49d3-kube-api-access-m757r\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:42:04.147468 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.147431 2560 generic.go:356] "Generic (PLEG): container finished" podID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerID="1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8" exitCode=0 Jul 29 14:42:04.147678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.147510 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" Jul 29 14:42:04.147678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.147515 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" event={"ID":"bc16ef99-cece-43a9-b95e-830676ef49d3","Type":"ContainerDied","Data":"1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8"} Jul 29 14:42:04.147678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.147615 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79" event={"ID":"bc16ef99-cece-43a9-b95e-830676ef49d3","Type":"ContainerDied","Data":"da8bd3bd4a75b081e1d5695e90517ae6134212b5ee41a93125f06bcf71f35e19"} Jul 29 14:42:04.147678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.147647 2560 scope.go:117] "RemoveContainer" containerID="1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8" Jul 29 14:42:04.158855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.158831 2560 scope.go:117] "RemoveContainer" containerID="99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21" Jul 29 14:42:04.168652 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.168630 2560 scope.go:117] "RemoveContainer" containerID="1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8" Jul 29 14:42:04.169020 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:42:04.168998 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8\": container with ID starting with 1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8 not found: ID does not exist" containerID="1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8" Jul 29 14:42:04.169108 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.169034 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8"} err="failed to get container status \"1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8\": rpc error: code = NotFound desc = could not find container \"1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8\": container with ID starting with 1ea88cf50d1cd4960478e7cccc7083a6cb536b8e6cf9a08ecaf4c9ab3be10db8 not found: ID does not exist" Jul 29 14:42:04.169108 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.169062 2560 scope.go:117] "RemoveContainer" containerID="99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21" Jul 29 14:42:04.169368 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:42:04.169352 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21\": container with ID starting with 99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21 not found: ID does not exist" containerID="99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21" Jul 29 14:42:04.169419 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.169375 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21"} err="failed to get container status \"99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21\": rpc error: code = NotFound desc = could not find container \"99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21\": container with ID starting with 99fd1d4e9d89df0e1780192900dc6cdcba69722d39406236959284dd360c3f21 not found: ID does not exist" Jul 29 14:42:04.171204 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.171185 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79"] Jul 29 14:42:04.174845 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:04.174824 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a20e0b-openshift-default-5c64dc995b-6vd79"] Jul 29 14:42:05.381791 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:05.381744 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" path="/var/lib/kubelet/pods/bc16ef99-cece-43a9-b95e-830676ef49d3/volumes" Jul 29 14:42:07.809170 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:07.809137 2560 scope.go:117] "RemoveContainer" containerID="647cdc0e0c3bc94a6d6028af7e3fe9de96c8a8f3b7bc2970d198f0990993fe40" Jul 29 14:42:14.379256 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:14.379206 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh"] Jul 29 14:42:14.382035 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:14.382001 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ffdf30-575zh"] Jul 29 14:42:15.381294 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:15.381257 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4845bcb-6891-418d-b150-92c3c4879dbe" path="/var/lib/kubelet/pods/c4845bcb-6891-418d-b150-92c3c4879dbe/volumes" Jul 29 14:42:19.251158 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:19.251121 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:42:21.896037 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.895994 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp"] Jul 29 14:42:21.896459 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.896446 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" Jul 29 14:42:21.896459 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.896459 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" Jul 29 14:42:21.896567 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.896479 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" Jul 29 14:42:21.896567 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.896485 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" Jul 29 14:42:21.896567 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.896495 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4845bcb-6891-418d-b150-92c3c4879dbe" containerName="revoke-keys" Jul 29 14:42:21.896567 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.896501 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4845bcb-6891-418d-b150-92c3c4879dbe" containerName="revoke-keys" Jul 29 14:42:21.896720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.896608 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4845bcb-6891-418d-b150-92c3c4879dbe" containerName="revoke-keys" Jul 29 14:42:21.896720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.896619 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" Jul 29 14:42:21.900142 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.900122 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:21.902515 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.902491 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-61bc0a-openshift-default-dockercfg-42kcb\"" Jul 29 14:42:21.913424 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.913397 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp"] Jul 29 14:42:21.944392 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.944355 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-istio-token\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:21.944585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.944400 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-credential-socket\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:21.944585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.944427 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629f2947-c498-4d52-923b-179e9bd599c8-istio-podinfo\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:21.944585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.944478 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-envoy\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:21.944585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.944513 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-data\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:21.944585 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.944560 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-certs\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:21.944763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.944602 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-socket\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:21.944763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.944636 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55tg\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-kube-api-access-n55tg\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:21.944763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:21.944659 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629f2947-c498-4d52-923b-179e9bd599c8-istiod-ca-cert\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.045579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045509 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-socket\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.045759 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045610 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55tg\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-kube-api-access-n55tg\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.045759 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045634 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629f2947-c498-4d52-923b-179e9bd599c8-istiod-ca-cert\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.045759 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-istio-token\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.045929 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045766 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-credential-socket\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.045929 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045818 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629f2947-c498-4d52-923b-179e9bd599c8-istio-podinfo\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.045929 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045852 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-envoy\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.045929 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045881 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-data\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.046137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045945 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-socket\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.046137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.045962 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-certs\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.046137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.046117 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-credential-socket\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.046419 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.046230 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-data\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.046419 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.046323 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-certs\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.046571 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.046520 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629f2947-c498-4d52-923b-179e9bd599c8-istiod-ca-cert\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.048187 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.048162 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-envoy\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.048828 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.048806 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629f2947-c498-4d52-923b-179e9bd599c8-istio-podinfo\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.054330 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.054299 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-istio-token\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.054460 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.054435 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55tg\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-kube-api-access-n55tg\") pod \"e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.212534 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.212492 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:22.365574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.365523 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp"] Jul 29 14:42:22.366290 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:42:22.366262 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629f2947_c498_4d52_923b_179e9bd599c8.slice/crio-096ad0be2493818e986028807bd2a288bf51ff979b52bbe6427206b4c5c86777 WatchSource:0}: Error finding container 096ad0be2493818e986028807bd2a288bf51ff979b52bbe6427206b4c5c86777: Status 404 returned error can't find the container with id 096ad0be2493818e986028807bd2a288bf51ff979b52bbe6427206b4c5c86777 Jul 29 14:42:22.368602 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.368563 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:42:22.368688 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.368652 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:42:22.368727 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:22.368695 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:42:23.273338 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:23.273306 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" event={"ID":"629f2947-c498-4d52-923b-179e9bd599c8","Type":"ContainerStarted","Data":"42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae"} Jul 29 14:42:23.273828 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:23.273357 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" event={"ID":"629f2947-c498-4d52-923b-179e9bd599c8","Type":"ContainerStarted","Data":"096ad0be2493818e986028807bd2a288bf51ff979b52bbe6427206b4c5c86777"} Jul 29 14:42:23.294288 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:23.294233 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" podStartSLOduration=2.294213431 podStartE2EDuration="2.294213431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:42:23.291929439 +0000 UTC m=+2476.493430641" watchObservedRunningTime="2026-07-29 14:42:23.294213431 +0000 UTC m=+2476.495714603" Jul 29 14:42:24.213007 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:24.212966 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:24.218213 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:24.218188 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:24.279459 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:24.279424 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:24.280394 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:24.280370 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:30.554511 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:30.554469 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:42:39.280790 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:39.280745 2560 patch_prober.go:28] interesting pod/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Jul 29 14:42:39.281216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:39.280817 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Jul 29 14:42:39.381075 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:39.381036 2560 generic.go:356] "Generic (PLEG): container finished" podID="629f2947-c498-4d52-923b-179e9bd599c8" containerID="42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae" exitCode=0 Jul 29 14:42:39.381746 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:39.381718 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" event={"ID":"629f2947-c498-4d52-923b-179e9bd599c8","Type":"ContainerDied","Data":"42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae"} Jul 29 14:42:39.382148 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:39.382131 2560 scope.go:117] "RemoveContainer" containerID="42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae" Jul 29 14:42:39.383261 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:39.383234 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:42:39.383333 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:39.383317 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:42:39.383593 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:39.383370 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:42:40.390364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:40.390316 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" event={"ID":"629f2947-c498-4d52-923b-179e9bd599c8","Type":"ContainerStarted","Data":"dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06"} Jul 29 14:42:41.212914 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:41.212871 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:41.213157 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:41.213135 2560 patch_prober.go:28] interesting pod/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Jul 29 14:42:41.213204 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:41.213189 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Jul 29 14:42:42.213805 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:42.213763 2560 patch_prober.go:28] interesting pod/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Jul 29 14:42:42.214231 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:42.213824 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Jul 29 14:42:43.213614 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:43.213574 2560 patch_prober.go:28] interesting pod/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Jul 29 14:42:43.213808 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:43.213635 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Jul 29 14:42:44.213388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:44.213338 2560 patch_prober.go:28] interesting pod/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Jul 29 14:42:44.213850 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:44.213407 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Jul 29 14:42:45.217415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:45.217376 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:45.217878 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:45.217757 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:45.218601 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:45.218581 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:42:54.210404 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.210357 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5"] Jul 29 14:42:54.211244 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.211223 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc16ef99-cece-43a9-b95e-830676ef49d3" containerName="istio-proxy" Jul 29 14:42:54.214954 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.214928 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.217512 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.217489 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-532c20-serving-cert\"" Jul 29 14:42:54.218841 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.218822 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jqfw\"" Jul 29 14:42:54.225841 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.225818 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5"] Jul 29 14:42:54.358833 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.358787 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kswp\" (UniqueName: \"kubernetes.io/projected/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-kube-api-access-9kswp\") pod \"maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.359022 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.358939 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.359067 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.359044 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.459946 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.459899 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.460161 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.459955 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kswp\" (UniqueName: \"kubernetes.io/projected/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-kube-api-access-9kswp\") pod \"maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.460161 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.460101 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.460849 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.460798 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.462642 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.462623 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.468383 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.468358 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kswp\" (UniqueName: \"kubernetes.io/projected/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-kube-api-access-9kswp\") pod \"maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.528086 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.528039 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:54.690565 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.690506 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5"] Jul 29 14:42:54.696048 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:42:54.696011 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae99d8b_2fee_494b_bd27_d60af6b8f4e5.slice/crio-bbd90d15fb6f92cf4613716fddbbf0808d8f3a103e8625c8e4ff93165acaa2f8 WatchSource:0}: Error finding container bbd90d15fb6f92cf4613716fddbbf0808d8f3a103e8625c8e4ff93165acaa2f8: Status 404 returned error can't find the container with id bbd90d15fb6f92cf4613716fddbbf0808d8f3a103e8625c8e4ff93165acaa2f8 Jul 29 14:42:54.697952 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:54.697924 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:42:55.512948 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:55.512857 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" event={"ID":"cae99d8b-2fee-494b-bd27-d60af6b8f4e5","Type":"ContainerStarted","Data":"9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298"} Jul 29 14:42:55.512948 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:55.512936 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" event={"ID":"cae99d8b-2fee-494b-bd27-d60af6b8f4e5","Type":"ContainerStarted","Data":"bbd90d15fb6f92cf4613716fddbbf0808d8f3a103e8625c8e4ff93165acaa2f8"} Jul 29 14:42:55.513368 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:55.513093 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:42:55.531390 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:42:55.531312 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" podStartSLOduration=1.049470253 podStartE2EDuration="1.531290503s" totalImagesPullingTime="481.82025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:42:55.529177614 +0000 UTC m=+2508.730678799" watchObservedRunningTime="2026-07-29 14:42:55.531290503 +0000 UTC m=+2508.732791675" Jul 29 14:43:01.524641 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:43:01.524532 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:43:02.550567 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:43:02.550501 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:43:38.353356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:43:38.353320 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:44:12.150769 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:44:12.150725 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:44:36.643889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:44:36.643790 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:45:00.170613 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.170576 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj"] Jul 29 14:45:00.174110 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.174091 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx"] Jul 29 14:45:00.174235 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.174218 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj" Jul 29 14:45:00.177510 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.177490 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk"] Jul 29 14:45:00.177643 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.177608 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx" Jul 29 14:45:00.178308 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.178288 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:45:00.180860 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.180795 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk" Jul 29 14:45:00.185884 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.185865 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj" Jul 29 14:45:00.193338 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.193320 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx" Jul 29 14:45:00.199948 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.199924 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj"] Jul 29 14:45:00.206510 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.206481 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk"] Jul 29 14:45:00.233793 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.233767 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk" Jul 29 14:45:00.241743 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.241709 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx"] Jul 29 14:45:00.399792 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.399766 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx"] Jul 29 14:45:00.402264 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:45:00.402221 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2032a639_972d_44eb_9339_807e9b5f8bc9.slice/crio-73f00193c868cb356bf339ddd72af56e6559a9c771ea4685dd3af9a0e75f9aa3 WatchSource:0}: Error finding container 73f00193c868cb356bf339ddd72af56e6559a9c771ea4685dd3af9a0e75f9aa3: Status 404 returned error can't find the container with id 73f00193c868cb356bf339ddd72af56e6559a9c771ea4685dd3af9a0e75f9aa3 Jul 29 14:45:00.428061 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.428030 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk"] Jul 29 14:45:00.428268 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:45:00.428246 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc651dc16_5f49_4ee3_888d_553eccfd48d4.slice/crio-56c76b6416c14d9b9a35652755fda3a318a08d0966bf90e86a11171fbddce344 WatchSource:0}: Error finding container 56c76b6416c14d9b9a35652755fda3a318a08d0966bf90e86a11171fbddce344: Status 404 returned error can't find the container with id 56c76b6416c14d9b9a35652755fda3a318a08d0966bf90e86a11171fbddce344 Jul 29 14:45:00.577821 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:00.577794 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj"] Jul 29 14:45:00.579942 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:45:00.579914 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2516203f_fde0_467e_824f_fbcdd8b20fdb.slice/crio-562c31b4472356aecb2498a0f1fd05820ddd8d63e9e08c7882fc5fc36426166a WatchSource:0}: Error finding container 562c31b4472356aecb2498a0f1fd05820ddd8d63e9e08c7882fc5fc36426166a: Status 404 returned error can't find the container with id 562c31b4472356aecb2498a0f1fd05820ddd8d63e9e08c7882fc5fc36426166a Jul 29 14:45:01.307327 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:01.307288 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj" event={"ID":"2516203f-fde0-467e-824f-fbcdd8b20fdb","Type":"ContainerStarted","Data":"19a62744b129cc14890a96d901a877280a6dbb6e91bb3e38ae3c44676e3be293"} Jul 29 14:45:01.307819 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:01.307354 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj" event={"ID":"2516203f-fde0-467e-824f-fbcdd8b20fdb","Type":"ContainerStarted","Data":"562c31b4472356aecb2498a0f1fd05820ddd8d63e9e08c7882fc5fc36426166a"} Jul 29 14:45:01.308942 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:01.308909 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk" event={"ID":"c651dc16-5f49-4ee3-888d-553eccfd48d4","Type":"ContainerStarted","Data":"903780d172de659a6a0b3d80106867db70a8740eaa0416fae7e0d29c264981f8"} Jul 29 14:45:01.309080 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:01.308951 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk" event={"ID":"c651dc16-5f49-4ee3-888d-553eccfd48d4","Type":"ContainerStarted","Data":"56c76b6416c14d9b9a35652755fda3a318a08d0966bf90e86a11171fbddce344"} Jul 29 14:45:01.310485 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:01.310457 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx" event={"ID":"2032a639-972d-44eb-9339-807e9b5f8bc9","Type":"ContainerStarted","Data":"f659172feff865c2f909e00c95a55298829a226707b0eb2cce122d7c1d21d04a"} Jul 29 14:45:01.310623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:01.310515 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx" event={"ID":"2032a639-972d-44eb-9339-807e9b5f8bc9","Type":"ContainerStarted","Data":"73f00193c868cb356bf339ddd72af56e6559a9c771ea4685dd3af9a0e75f9aa3"} Jul 29 14:45:01.328562 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:01.328494 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj" podStartSLOduration=1.328478329 podStartE2EDuration="1.328478329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:45:01.325771291 +0000 UTC m=+2634.527272466" watchObservedRunningTime="2026-07-29 14:45:01.328478329 +0000 UTC m=+2634.529979571" Jul 29 14:45:01.342437 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:01.342380 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx" podStartSLOduration=1.342363903 podStartE2EDuration="1.342363903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:45:01.342170454 +0000 UTC m=+2634.543671626" watchObservedRunningTime="2026-07-29 14:45:01.342363903 +0000 UTC m=+2634.543865075" Jul 29 14:45:01.358382 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:01.358333 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk" podStartSLOduration=1.358318543 podStartE2EDuration="1.358318543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:45:01.357223812 +0000 UTC m=+2634.558724984" watchObservedRunningTime="2026-07-29 14:45:01.358318543 +0000 UTC m=+2634.559819715" Jul 29 14:45:02.327047 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:02.327009 2560 generic.go:356] "Generic (PLEG): container finished" podID="2516203f-fde0-467e-824f-fbcdd8b20fdb" containerID="19a62744b129cc14890a96d901a877280a6dbb6e91bb3e38ae3c44676e3be293" exitCode=0 Jul 29 14:45:02.327487 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:02.327089 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj" event={"ID":"2516203f-fde0-467e-824f-fbcdd8b20fdb","Type":"ContainerDied","Data":"19a62744b129cc14890a96d901a877280a6dbb6e91bb3e38ae3c44676e3be293"} Jul 29 14:45:02.328834 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:02.328809 2560 generic.go:356] "Generic (PLEG): container finished" podID="c651dc16-5f49-4ee3-888d-553eccfd48d4" containerID="903780d172de659a6a0b3d80106867db70a8740eaa0416fae7e0d29c264981f8" exitCode=0 Jul 29 14:45:02.328964 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:02.328896 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk" event={"ID":"c651dc16-5f49-4ee3-888d-553eccfd48d4","Type":"ContainerDied","Data":"903780d172de659a6a0b3d80106867db70a8740eaa0416fae7e0d29c264981f8"} Jul 29 14:45:02.330504 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:02.330483 2560 generic.go:356] "Generic (PLEG): container finished" podID="2032a639-972d-44eb-9339-807e9b5f8bc9" containerID="f659172feff865c2f909e00c95a55298829a226707b0eb2cce122d7c1d21d04a" exitCode=0 Jul 29 14:45:02.330635 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:02.330579 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx" event={"ID":"2032a639-972d-44eb-9339-807e9b5f8bc9","Type":"ContainerDied","Data":"f659172feff865c2f909e00c95a55298829a226707b0eb2cce122d7c1d21d04a"} Jul 29 14:45:03.495042 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:03.494904 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk" Jul 29 14:45:03.536245 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:03.536216 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx" Jul 29 14:45:03.552378 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:03.552346 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj" Jul 29 14:45:03.645794 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:03.645700 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:45:04.345116 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:04.345082 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj" Jul 29 14:45:04.345116 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:04.345099 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj" event={"ID":"2516203f-fde0-467e-824f-fbcdd8b20fdb","Type":"ContainerDied","Data":"562c31b4472356aecb2498a0f1fd05820ddd8d63e9e08c7882fc5fc36426166a"} Jul 29 14:45:04.345373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:04.345151 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="562c31b4472356aecb2498a0f1fd05820ddd8d63e9e08c7882fc5fc36426166a" Jul 29 14:45:04.346743 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:04.346720 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk" Jul 29 14:45:04.346895 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:04.346734 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk" event={"ID":"c651dc16-5f49-4ee3-888d-553eccfd48d4","Type":"ContainerDied","Data":"56c76b6416c14d9b9a35652755fda3a318a08d0966bf90e86a11171fbddce344"} Jul 29 14:45:04.346895 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:04.346786 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c76b6416c14d9b9a35652755fda3a318a08d0966bf90e86a11171fbddce344" Jul 29 14:45:04.348250 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:04.348231 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx" Jul 29 14:45:04.348373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:04.348251 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755605-frxbx" event={"ID":"2032a639-972d-44eb-9339-807e9b5f8bc9","Type":"ContainerDied","Data":"73f00193c868cb356bf339ddd72af56e6559a9c771ea4685dd3af9a0e75f9aa3"} Jul 29 14:45:04.348373 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:04.348291 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f00193c868cb356bf339ddd72af56e6559a9c771ea4685dd3af9a0e75f9aa3" Jul 29 14:45:22.849176 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:22.849137 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:45:49.462040 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:45:49.461995 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:46:07.646795 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:46:07.646766 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:46:07.650630 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:46:07.650606 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:46:07.673151 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:46:07.673125 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:46:07.676758 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:46:07.676735 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:46:07.870859 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:46:07.870827 2560 scope.go:117] "RemoveContainer" containerID="68873419f4fa5e1d753c8354f27ae5d89cbc9990fe8e3c6d50473ba4bb0f4574" Jul 29 14:46:08.066571 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:46:08.066502 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:46:34.548319 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:46:34.548275 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:46:54.048530 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:46:54.048490 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:47:07.899085 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:47:07.899049 2560 scope.go:117] "RemoveContainer" containerID="4c22a58c3f3143bfbeb2a080ba8c035367dd4a451927da624a58a4bd541fc301" Jul 29 14:47:20.460554 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:47:20.460494 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:47:37.863062 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:47:37.863024 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:48:05.054882 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:48:05.054843 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:48:07.923556 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:48:07.923506 2560 scope.go:117] "RemoveContainer" containerID="c598b3ca5885cdad499bf6ed87181d3ce26cff73de7c261697ad183b1355f208" Jul 29 14:48:07.933488 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:48:07.933466 2560 scope.go:117] "RemoveContainer" containerID="aa1e3bc83f45b0736c680e5e4ec50d8367635218d14532ef75fbb890bc9c23ff" Jul 29 14:48:23.643788 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:48:23.643739 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:48:47.073968 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:48:47.073924 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:49:03.548178 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:49:03.548136 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:49:13.156797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:49:13.156752 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:49:37.997254 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:49:37.997218 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:50:11.554609 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:50:11.554525 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:50:32.353025 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:50:32.352930 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:50:50.653269 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:50:50.653233 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:51:07.689068 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:51:07.689036 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:51:07.702977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:51:07.702950 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:51:07.724272 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:51:07.724239 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:51:07.727756 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:51:07.727716 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:51:13.057739 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:51:13.057705 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:51:58.648647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:51:58.648551 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:52:17.350334 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:17.350290 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:52:36.642778 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:36.642738 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:52:45.781262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781226 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7620-140194059129392"] Jul 29 14:52:45.781720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781688 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2516203f-fde0-467e-824f-fbcdd8b20fdb" containerName="cleanup" Jul 29 14:52:45.781720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781701 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2516203f-fde0-467e-824f-fbcdd8b20fdb" containerName="cleanup" Jul 29 14:52:45.781720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781712 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c651dc16-5f49-4ee3-888d-553eccfd48d4" containerName="cleanup" Jul 29 14:52:45.781720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781718 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c651dc16-5f49-4ee3-888d-553eccfd48d4" containerName="cleanup" Jul 29 14:52:45.781890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781730 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2032a639-972d-44eb-9339-807e9b5f8bc9" containerName="cleanup" Jul 29 14:52:45.781890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781735 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2032a639-972d-44eb-9339-807e9b5f8bc9" containerName="cleanup" Jul 29 14:52:45.781890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781811 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2032a639-972d-44eb-9339-807e9b5f8bc9" containerName="cleanup" Jul 29 14:52:45.781890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781819 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c651dc16-5f49-4ee3-888d-553eccfd48d4" containerName="cleanup" Jul 29 14:52:45.781890 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.781827 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2516203f-fde0-467e-824f-fbcdd8b20fdb" containerName="cleanup" Jul 29 14:52:45.784944 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.784928 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194059129392" Jul 29 14:52:45.787722 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.787677 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kq7nh\"" Jul 29 14:52:45.791394 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.791363 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194059129392"] Jul 29 14:52:45.900090 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:45.900048 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lm2p\" (UniqueName: \"kubernetes.io/projected/d038a904-c141-4178-8be4-2dda883d27ed-kube-api-access-6lm2p\") pod \"test-curl-7620-140194059129392\" (UID: \"d038a904-c141-4178-8be4-2dda883d27ed\") " pod="opendatahub/test-curl-7620-140194059129392" Jul 29 14:52:46.001570 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:46.001499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lm2p\" (UniqueName: \"kubernetes.io/projected/d038a904-c141-4178-8be4-2dda883d27ed-kube-api-access-6lm2p\") pod \"test-curl-7620-140194059129392\" (UID: \"d038a904-c141-4178-8be4-2dda883d27ed\") " pod="opendatahub/test-curl-7620-140194059129392" Jul 29 14:52:46.009290 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:46.009262 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lm2p\" (UniqueName: \"kubernetes.io/projected/d038a904-c141-4178-8be4-2dda883d27ed-kube-api-access-6lm2p\") pod \"test-curl-7620-140194059129392\" (UID: \"d038a904-c141-4178-8be4-2dda883d27ed\") " pod="opendatahub/test-curl-7620-140194059129392" Jul 29 14:52:46.097164 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:46.097067 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194059129392" Jul 29 14:52:46.241403 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:46.241376 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194059129392"] Jul 29 14:52:46.243425 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:52:46.243392 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd038a904_c141_4178_8be4_2dda883d27ed.slice/crio-5d4c61dc8076d08c63dbee2603f276046dfc0a5a8b8995acffe1b848c6e9b4d6 WatchSource:0}: Error finding container 5d4c61dc8076d08c63dbee2603f276046dfc0a5a8b8995acffe1b848c6e9b4d6: Status 404 returned error can't find the container with id 5d4c61dc8076d08c63dbee2603f276046dfc0a5a8b8995acffe1b848c6e9b4d6 Jul 29 14:52:46.244832 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:46.244816 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:52:46.331589 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:46.331531 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194059129392" event={"ID":"d038a904-c141-4178-8be4-2dda883d27ed","Type":"ContainerStarted","Data":"5d4c61dc8076d08c63dbee2603f276046dfc0a5a8b8995acffe1b848c6e9b4d6"} Jul 29 14:52:47.338922 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:47.338827 2560 generic.go:356] "Generic (PLEG): container finished" podID="d038a904-c141-4178-8be4-2dda883d27ed" containerID="e6acd6233bb5e7af4c075534d9d3479e0fdb8e7b7f4645a3d84a3bb45bc27d30" exitCode=0 Jul 29 14:52:47.339300 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:47.338913 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194059129392" event={"ID":"d038a904-c141-4178-8be4-2dda883d27ed","Type":"ContainerDied","Data":"e6acd6233bb5e7af4c075534d9d3479e0fdb8e7b7f4645a3d84a3bb45bc27d30"} Jul 29 14:52:48.501287 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.501255 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194059129392" Jul 29 14:52:48.536211 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.536173 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7620-140194059129392_d038a904-c141-4178-8be4-2dda883d27ed/test-curl-7620-140194059129392/0.log" Jul 29 14:52:48.563083 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.563045 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7620-140194059129392"] Jul 29 14:52:48.567264 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.567238 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7620-140194059129392"] Jul 29 14:52:48.627703 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.627608 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lm2p\" (UniqueName: \"kubernetes.io/projected/d038a904-c141-4178-8be4-2dda883d27ed-kube-api-access-6lm2p\") pod \"d038a904-c141-4178-8be4-2dda883d27ed\" (UID: \"d038a904-c141-4178-8be4-2dda883d27ed\") " Jul 29 14:52:48.630136 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.630111 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d038a904-c141-4178-8be4-2dda883d27ed-kube-api-access-6lm2p" (OuterVolumeSpecName: "kube-api-access-6lm2p") pod "d038a904-c141-4178-8be4-2dda883d27ed" (UID: "d038a904-c141-4178-8be4-2dda883d27ed"). InnerVolumeSpecName "kube-api-access-6lm2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:52:48.712872 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.712834 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7620-140194059100592"] Jul 29 14:52:48.713325 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.713311 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d038a904-c141-4178-8be4-2dda883d27ed" containerName="test-curl-7620-140194059129392" Jul 29 14:52:48.713370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.713326 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d038a904-c141-4178-8be4-2dda883d27ed" containerName="test-curl-7620-140194059129392" Jul 29 14:52:48.713406 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.713401 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d038a904-c141-4178-8be4-2dda883d27ed" containerName="test-curl-7620-140194059129392" Jul 29 14:52:48.716621 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.716606 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194059100592" Jul 29 14:52:48.722804 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.722776 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194059100592"] Jul 29 14:52:48.729032 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.729002 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lm2p\" (UniqueName: \"kubernetes.io/projected/d038a904-c141-4178-8be4-2dda883d27ed-kube-api-access-6lm2p\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:52:48.829973 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.829927 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5z66\" (UniqueName: \"kubernetes.io/projected/374d63a3-55b9-4653-8f7c-2115b604b3c3-kube-api-access-v5z66\") pod \"test-curl-7620-140194059100592\" (UID: \"374d63a3-55b9-4653-8f7c-2115b604b3c3\") " pod="opendatahub/test-curl-7620-140194059100592" Jul 29 14:52:48.931152 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.931116 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5z66\" (UniqueName: \"kubernetes.io/projected/374d63a3-55b9-4653-8f7c-2115b604b3c3-kube-api-access-v5z66\") pod \"test-curl-7620-140194059100592\" (UID: \"374d63a3-55b9-4653-8f7c-2115b604b3c3\") " pod="opendatahub/test-curl-7620-140194059100592" Jul 29 14:52:48.939352 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.939314 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5z66\" (UniqueName: \"kubernetes.io/projected/374d63a3-55b9-4653-8f7c-2115b604b3c3-kube-api-access-v5z66\") pod \"test-curl-7620-140194059100592\" (UID: \"374d63a3-55b9-4653-8f7c-2115b604b3c3\") " pod="opendatahub/test-curl-7620-140194059100592" Jul 29 14:52:48.992662 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.992624 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n5hbk"] Jul 29 14:52:48.996492 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:48.996467 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:52:49.004902 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.004871 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n5hbk"] Jul 29 14:52:49.027079 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.027039 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194059100592" Jul 29 14:52:49.133439 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.133399 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q66d\" (UniqueName: \"kubernetes.io/projected/00542df7-5678-40c5-812d-50a32517898a-kube-api-access-9q66d\") pod \"kuadrant-operator-catalog-n5hbk\" (UID: \"00542df7-5678-40c5-812d-50a32517898a\") " pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:52:49.173686 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.173657 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194059100592"] Jul 29 14:52:49.175533 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:52:49.175511 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod374d63a3_55b9_4653_8f7c_2115b604b3c3.slice/crio-6e9588d2cba78224aec661cef8d33c49eb502b90a126866196e5d40adfd7edae WatchSource:0}: Error finding container 6e9588d2cba78224aec661cef8d33c49eb502b90a126866196e5d40adfd7edae: Status 404 returned error can't find the container with id 6e9588d2cba78224aec661cef8d33c49eb502b90a126866196e5d40adfd7edae Jul 29 14:52:49.234956 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.234913 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q66d\" (UniqueName: \"kubernetes.io/projected/00542df7-5678-40c5-812d-50a32517898a-kube-api-access-9q66d\") pod \"kuadrant-operator-catalog-n5hbk\" (UID: \"00542df7-5678-40c5-812d-50a32517898a\") " pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:52:49.244144 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.244120 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q66d\" (UniqueName: \"kubernetes.io/projected/00542df7-5678-40c5-812d-50a32517898a-kube-api-access-9q66d\") pod \"kuadrant-operator-catalog-n5hbk\" (UID: \"00542df7-5678-40c5-812d-50a32517898a\") " pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:52:49.310241 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.310199 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:52:49.353574 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.353525 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d4c61dc8076d08c63dbee2603f276046dfc0a5a8b8995acffe1b848c6e9b4d6" Jul 29 14:52:49.353756 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.353588 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194059129392" Jul 29 14:52:49.354878 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.354852 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194059100592" event={"ID":"374d63a3-55b9-4653-8f7c-2115b604b3c3","Type":"ContainerStarted","Data":"6e9588d2cba78224aec661cef8d33c49eb502b90a126866196e5d40adfd7edae"} Jul 29 14:52:49.368141 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:52:49.368112 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7620-140194059129392\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="d038a904-c141-4178-8be4-2dda883d27ed" pod="opendatahub/test-curl-7620-140194059129392" Jul 29 14:52:49.386425 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.386332 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d038a904-c141-4178-8be4-2dda883d27ed" path="/var/lib/kubelet/pods/d038a904-c141-4178-8be4-2dda883d27ed/volumes" Jul 29 14:52:49.465102 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:49.465081 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n5hbk"] Jul 29 14:52:49.469506 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:52:49.469479 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00542df7_5678_40c5_812d_50a32517898a.slice/crio-3fd6b47b46dbdd983be1d2afba64f41965e358ac9687737095dfaf51b62aa39d WatchSource:0}: Error finding container 3fd6b47b46dbdd983be1d2afba64f41965e358ac9687737095dfaf51b62aa39d: Status 404 returned error can't find the container with id 3fd6b47b46dbdd983be1d2afba64f41965e358ac9687737095dfaf51b62aa39d Jul 29 14:52:50.362948 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:50.362912 2560 generic.go:356] "Generic (PLEG): container finished" podID="374d63a3-55b9-4653-8f7c-2115b604b3c3" containerID="b5f8fa3b60fd726c132002f98ff9001bfbfb4ff6beffb4085da051b43b08540a" exitCode=0 Jul 29 14:52:50.363387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:50.362989 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194059100592" event={"ID":"374d63a3-55b9-4653-8f7c-2115b604b3c3","Type":"ContainerDied","Data":"b5f8fa3b60fd726c132002f98ff9001bfbfb4ff6beffb4085da051b43b08540a"} Jul 29 14:52:50.364789 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:50.364760 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" event={"ID":"00542df7-5678-40c5-812d-50a32517898a","Type":"ContainerStarted","Data":"fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f"} Jul 29 14:52:50.364906 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:50.364815 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" event={"ID":"00542df7-5678-40c5-812d-50a32517898a","Type":"ContainerStarted","Data":"3fd6b47b46dbdd983be1d2afba64f41965e358ac9687737095dfaf51b62aa39d"} Jul 29 14:52:50.395195 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:50.395126 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" podStartSLOduration=1.648692653 podStartE2EDuration="2.395105569s" totalImagesPullingTime="746.412916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:52:50.393706852 +0000 UTC m=+3103.595208026" watchObservedRunningTime="2026-07-29 14:52:50.395105569 +0000 UTC m=+3103.596606747" Jul 29 14:52:51.048723 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.048677 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:52:51.522270 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.522243 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194059100592" Jul 29 14:52:51.542788 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.542755 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7620-140194059100592_374d63a3-55b9-4653-8f7c-2115b604b3c3/test-curl-7620-140194059100592/0.log" Jul 29 14:52:51.565840 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.565791 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7620-140194059100592"] Jul 29 14:52:51.568557 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.568514 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7620-140194059100592"] Jul 29 14:52:51.659580 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.659437 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5z66\" (UniqueName: \"kubernetes.io/projected/374d63a3-55b9-4653-8f7c-2115b604b3c3-kube-api-access-v5z66\") pod \"374d63a3-55b9-4653-8f7c-2115b604b3c3\" (UID: \"374d63a3-55b9-4653-8f7c-2115b604b3c3\") " Jul 29 14:52:51.661745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.661713 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374d63a3-55b9-4653-8f7c-2115b604b3c3-kube-api-access-v5z66" (OuterVolumeSpecName: "kube-api-access-v5z66") pod "374d63a3-55b9-4653-8f7c-2115b604b3c3" (UID: "374d63a3-55b9-4653-8f7c-2115b604b3c3"). InnerVolumeSpecName "kube-api-access-v5z66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:52:51.747781 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.747745 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:51.748211 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.748197 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374d63a3-55b9-4653-8f7c-2115b604b3c3" containerName="test-curl-7620-140194059100592" Jul 29 14:52:51.748256 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.748213 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="374d63a3-55b9-4653-8f7c-2115b604b3c3" containerName="test-curl-7620-140194059100592" Jul 29 14:52:51.748296 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.748276 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="374d63a3-55b9-4653-8f7c-2115b604b3c3" containerName="test-curl-7620-140194059100592" Jul 29 14:52:51.757921 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.757284 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:51.760831 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.760798 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5z66\" (UniqueName: \"kubernetes.io/projected/374d63a3-55b9-4653-8f7c-2115b604b3c3-kube-api-access-v5z66\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:52:51.760996 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.760829 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:51.861629 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.861588 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvjh\" (UniqueName: \"kubernetes.io/projected/b1901403-93d7-492b-ba92-d7ca509e1653-kube-api-access-2jvjh\") pod \"test-curl-7620-140194058983440\" (UID: \"b1901403-93d7-492b-ba92-d7ca509e1653\") " pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:51.963016 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.962978 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvjh\" (UniqueName: \"kubernetes.io/projected/b1901403-93d7-492b-ba92-d7ca509e1653-kube-api-access-2jvjh\") pod \"test-curl-7620-140194058983440\" (UID: \"b1901403-93d7-492b-ba92-d7ca509e1653\") " pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:51.970974 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:51.970943 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvjh\" (UniqueName: \"kubernetes.io/projected/b1901403-93d7-492b-ba92-d7ca509e1653-kube-api-access-2jvjh\") pod \"test-curl-7620-140194058983440\" (UID: \"b1901403-93d7-492b-ba92-d7ca509e1653\") " pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:52.069889 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:52.069850 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:52.219517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:52.219483 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:52.220362 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:52:52.220338 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1901403_93d7_492b_ba92_d7ca509e1653.slice/crio-8c16b40383538d26cd23ecdc5ef3f4cc7b08b56afd20c8aae46e0d35921b7a14 WatchSource:0}: Error finding container 8c16b40383538d26cd23ecdc5ef3f4cc7b08b56afd20c8aae46e0d35921b7a14: Status 404 returned error can't find the container with id 8c16b40383538d26cd23ecdc5ef3f4cc7b08b56afd20c8aae46e0d35921b7a14 Jul 29 14:52:52.378713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:52.378683 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9588d2cba78224aec661cef8d33c49eb502b90a126866196e5d40adfd7edae" Jul 29 14:52:52.378713 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:52.378703 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194059100592" Jul 29 14:52:52.380123 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:52.380095 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194058983440" event={"ID":"b1901403-93d7-492b-ba92-d7ca509e1653","Type":"ContainerStarted","Data":"8c16b40383538d26cd23ecdc5ef3f4cc7b08b56afd20c8aae46e0d35921b7a14"} Jul 29 14:52:52.389424 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:52:52.389394 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7620-140194059100592\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="374d63a3-55b9-4653-8f7c-2115b604b3c3" pod="opendatahub/test-curl-7620-140194059100592" Jul 29 14:52:53.382514 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:53.382475 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="374d63a3-55b9-4653-8f7c-2115b604b3c3" path="/var/lib/kubelet/pods/374d63a3-55b9-4653-8f7c-2115b604b3c3/volumes" Jul 29 14:52:53.387432 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:53.387404 2560 generic.go:356] "Generic (PLEG): container finished" podID="b1901403-93d7-492b-ba92-d7ca509e1653" containerID="ad3260f78d0dbff7828adfd6b2f367815ec3ce2c22a5de82be110ff7b4e9770f" exitCode=0 Jul 29 14:52:53.387622 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:53.387475 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194058983440" event={"ID":"b1901403-93d7-492b-ba92-d7ca509e1653","Type":"ContainerDied","Data":"ad3260f78d0dbff7828adfd6b2f367815ec3ce2c22a5de82be110ff7b4e9770f"} Jul 29 14:52:54.549745 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.549712 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:54.572343 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.572315 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7620-140194058983440_b1901403-93d7-492b-ba92-d7ca509e1653/test-curl-7620-140194058983440/0.log" Jul 29 14:52:54.600051 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.600017 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:54.605626 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.605598 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:54.689065 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.689030 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jvjh\" (UniqueName: \"kubernetes.io/projected/b1901403-93d7-492b-ba92-d7ca509e1653-kube-api-access-2jvjh\") pod \"b1901403-93d7-492b-ba92-d7ca509e1653\" (UID: \"b1901403-93d7-492b-ba92-d7ca509e1653\") " Jul 29 14:52:54.691367 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.691339 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1901403-93d7-492b-ba92-d7ca509e1653-kube-api-access-2jvjh" (OuterVolumeSpecName: "kube-api-access-2jvjh") pod "b1901403-93d7-492b-ba92-d7ca509e1653" (UID: "b1901403-93d7-492b-ba92-d7ca509e1653"). InnerVolumeSpecName "kube-api-access-2jvjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:52:54.728641 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.728597 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:54.729130 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.729112 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1901403-93d7-492b-ba92-d7ca509e1653" containerName="test-curl-7620-140194058983440" Jul 29 14:52:54.729229 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.729134 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1901403-93d7-492b-ba92-d7ca509e1653" containerName="test-curl-7620-140194058983440" Jul 29 14:52:54.729293 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.729280 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1901403-93d7-492b-ba92-d7ca509e1653" containerName="test-curl-7620-140194058983440" Jul 29 14:52:54.732588 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.732531 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:54.741032 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.741003 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:54.790914 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.790877 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvjh\" (UniqueName: \"kubernetes.io/projected/b1901403-93d7-492b-ba92-d7ca509e1653-kube-api-access-2jvjh\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:52:54.891863 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.891826 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hfm2\" (UniqueName: \"kubernetes.io/projected/a1fc8914-2862-4d98-80fa-ff6043c63a34-kube-api-access-6hfm2\") pod \"test-curl-7620-140194058983440\" (UID: \"a1fc8914-2862-4d98-80fa-ff6043c63a34\") " pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:54.993209 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:54.993112 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hfm2\" (UniqueName: \"kubernetes.io/projected/a1fc8914-2862-4d98-80fa-ff6043c63a34-kube-api-access-6hfm2\") pod \"test-curl-7620-140194058983440\" (UID: \"a1fc8914-2862-4d98-80fa-ff6043c63a34\") " pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:55.003314 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:55.003287 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hfm2\" (UniqueName: \"kubernetes.io/projected/a1fc8914-2862-4d98-80fa-ff6043c63a34-kube-api-access-6hfm2\") pod \"test-curl-7620-140194058983440\" (UID: \"a1fc8914-2862-4d98-80fa-ff6043c63a34\") " pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:55.044309 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:55.044255 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:55.199877 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:55.199854 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:55.201579 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:52:55.201504 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1fc8914_2862_4d98_80fa_ff6043c63a34.slice/crio-2be3d7069ede9a04bb7744b2f101b02502ee47513419ace47ac408fa7c31d084 WatchSource:0}: Error finding container 2be3d7069ede9a04bb7744b2f101b02502ee47513419ace47ac408fa7c31d084: Status 404 returned error can't find the container with id 2be3d7069ede9a04bb7744b2f101b02502ee47513419ace47ac408fa7c31d084 Jul 29 14:52:55.381912 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:55.381830 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1901403-93d7-492b-ba92-d7ca509e1653" path="/var/lib/kubelet/pods/b1901403-93d7-492b-ba92-d7ca509e1653/volumes" Jul 29 14:52:55.403579 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:55.403532 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:55.403770 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:55.403586 2560 scope.go:117] "RemoveContainer" containerID="ad3260f78d0dbff7828adfd6b2f367815ec3ce2c22a5de82be110ff7b4e9770f" Jul 29 14:52:55.405190 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:55.405162 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194058983440" event={"ID":"a1fc8914-2862-4d98-80fa-ff6043c63a34","Type":"ContainerStarted","Data":"2be3d7069ede9a04bb7744b2f101b02502ee47513419ace47ac408fa7c31d084"} Jul 29 14:52:56.415972 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:56.415934 2560 generic.go:356] "Generic (PLEG): container finished" podID="a1fc8914-2862-4d98-80fa-ff6043c63a34" containerID="0361141d7d3efb24e5ad0edaa9b8a1b68f57d7f43cdae1c657a9e90e25bbf397" exitCode=0 Jul 29 14:52:56.416390 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:56.416033 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194058983440" event={"ID":"a1fc8914-2862-4d98-80fa-ff6043c63a34","Type":"ContainerDied","Data":"0361141d7d3efb24e5ad0edaa9b8a1b68f57d7f43cdae1c657a9e90e25bbf397"} Jul 29 14:52:57.570605 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.570568 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:57.590766 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.590733 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7620-140194058983440_a1fc8914-2862-4d98-80fa-ff6043c63a34/test-curl-7620-140194058983440/0.log" Jul 29 14:52:57.618287 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.618240 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:57.622958 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.622927 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7620-140194058983440"] Jul 29 14:52:57.721866 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.721825 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hfm2\" (UniqueName: \"kubernetes.io/projected/a1fc8914-2862-4d98-80fa-ff6043c63a34-kube-api-access-6hfm2\") pod \"a1fc8914-2862-4d98-80fa-ff6043c63a34\" (UID: \"a1fc8914-2862-4d98-80fa-ff6043c63a34\") " Jul 29 14:52:57.724120 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.724085 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fc8914-2862-4d98-80fa-ff6043c63a34-kube-api-access-6hfm2" (OuterVolumeSpecName: "kube-api-access-6hfm2") pod "a1fc8914-2862-4d98-80fa-ff6043c63a34" (UID: "a1fc8914-2862-4d98-80fa-ff6043c63a34"). InnerVolumeSpecName "kube-api-access-6hfm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:52:57.764586 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.764533 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:52:57.765047 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.765032 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1fc8914-2862-4d98-80fa-ff6043c63a34" containerName="test-curl-7620-140194058983440" Jul 29 14:52:57.765093 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.765050 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1fc8914-2862-4d98-80fa-ff6043c63a34" containerName="test-curl-7620-140194058983440" Jul 29 14:52:57.765136 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.765127 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1fc8914-2862-4d98-80fa-ff6043c63a34" containerName="test-curl-7620-140194058983440" Jul 29 14:52:57.768401 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.768383 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:52:57.775739 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.775674 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:52:57.822998 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.822960 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hfm2\" (UniqueName: \"kubernetes.io/projected/a1fc8914-2862-4d98-80fa-ff6043c63a34-kube-api-access-6hfm2\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:52:57.923977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:57.923938 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fhgp\" (UniqueName: \"kubernetes.io/projected/7cdd012d-254d-444d-8757-216ee33727c2-kube-api-access-6fhgp\") pod \"test-curl-7620-140194058984448\" (UID: \"7cdd012d-254d-444d-8757-216ee33727c2\") " pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:52:58.025493 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:58.025401 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fhgp\" (UniqueName: \"kubernetes.io/projected/7cdd012d-254d-444d-8757-216ee33727c2-kube-api-access-6fhgp\") pod \"test-curl-7620-140194058984448\" (UID: \"7cdd012d-254d-444d-8757-216ee33727c2\") " pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:52:58.034466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:58.034433 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fhgp\" (UniqueName: \"kubernetes.io/projected/7cdd012d-254d-444d-8757-216ee33727c2-kube-api-access-6fhgp\") pod \"test-curl-7620-140194058984448\" (UID: \"7cdd012d-254d-444d-8757-216ee33727c2\") " pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:52:58.080675 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:58.080634 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:52:58.230825 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:58.230796 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:52:58.232183 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:52:58.232156 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cdd012d_254d_444d_8757_216ee33727c2.slice/crio-6fde9ebe7f8c7f99d476ffb557c9a8bc00285d9cf50c7d007fa9dec506c43b55 WatchSource:0}: Error finding container 6fde9ebe7f8c7f99d476ffb557c9a8bc00285d9cf50c7d007fa9dec506c43b55: Status 404 returned error can't find the container with id 6fde9ebe7f8c7f99d476ffb557c9a8bc00285d9cf50c7d007fa9dec506c43b55 Jul 29 14:52:58.430380 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:58.430352 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be3d7069ede9a04bb7744b2f101b02502ee47513419ace47ac408fa7c31d084" Jul 29 14:52:58.430380 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:58.430361 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:58.431834 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:58.431806 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194058984448" event={"ID":"7cdd012d-254d-444d-8757-216ee33727c2","Type":"ContainerStarted","Data":"6fde9ebe7f8c7f99d476ffb557c9a8bc00285d9cf50c7d007fa9dec506c43b55"} Jul 29 14:52:58.441460 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:52:58.441430 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7620-140194058983440\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="a1fc8914-2862-4d98-80fa-ff6043c63a34" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:59.310548 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:59.310504 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:52:59.310957 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:59.310586 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:52:59.333641 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:59.333614 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:52:59.336143 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:52:59.336108 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7620-140194058983440\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="a1fc8914-2862-4d98-80fa-ff6043c63a34" pod="opendatahub/test-curl-7620-140194058983440" Jul 29 14:52:59.381571 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:59.381520 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1fc8914-2862-4d98-80fa-ff6043c63a34" path="/var/lib/kubelet/pods/a1fc8914-2862-4d98-80fa-ff6043c63a34/volumes" Jul 29 14:52:59.440144 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:59.440108 2560 generic.go:356] "Generic (PLEG): container finished" podID="7cdd012d-254d-444d-8757-216ee33727c2" containerID="523cdd136c6170257771b9bddd22f7a144278448750f231631f07e16b48c0d01" exitCode=0 Jul 29 14:52:59.440305 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:59.440197 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194058984448" event={"ID":"7cdd012d-254d-444d-8757-216ee33727c2","Type":"ContainerDied","Data":"523cdd136c6170257771b9bddd22f7a144278448750f231631f07e16b48c0d01"} Jul 29 14:52:59.464446 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:59.464418 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:52:59.563080 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:52:59.562985 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n5hbk"] Jul 29 14:53:00.597740 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.597714 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:00.617101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.617074 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7620-140194058984448_7cdd012d-254d-444d-8757-216ee33727c2/test-curl-7620-140194058984448/0.log" Jul 29 14:53:00.646888 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.646845 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:53:00.652753 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.652712 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:53:00.753034 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.752938 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fhgp\" (UniqueName: \"kubernetes.io/projected/7cdd012d-254d-444d-8757-216ee33727c2-kube-api-access-6fhgp\") pod \"7cdd012d-254d-444d-8757-216ee33727c2\" (UID: \"7cdd012d-254d-444d-8757-216ee33727c2\") " Jul 29 14:53:00.755293 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.755261 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdd012d-254d-444d-8757-216ee33727c2-kube-api-access-6fhgp" (OuterVolumeSpecName: "kube-api-access-6fhgp") pod "7cdd012d-254d-444d-8757-216ee33727c2" (UID: "7cdd012d-254d-444d-8757-216ee33727c2"). InnerVolumeSpecName "kube-api-access-6fhgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:53:00.784735 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.784694 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:53:00.785216 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.785201 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cdd012d-254d-444d-8757-216ee33727c2" containerName="test-curl-7620-140194058984448" Jul 29 14:53:00.785267 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.785219 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cdd012d-254d-444d-8757-216ee33727c2" containerName="test-curl-7620-140194058984448" Jul 29 14:53:00.785299 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.785286 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cdd012d-254d-444d-8757-216ee33727c2" containerName="test-curl-7620-140194058984448" Jul 29 14:53:00.788521 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.788502 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:00.796255 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.796224 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:53:00.854617 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.854574 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fhgp\" (UniqueName: \"kubernetes.io/projected/7cdd012d-254d-444d-8757-216ee33727c2-kube-api-access-6fhgp\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:53:00.956039 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:00.955993 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96bj7\" (UniqueName: \"kubernetes.io/projected/f4ce1fc4-058d-4782-8c43-3df376b68a34-kube-api-access-96bj7\") pod \"test-curl-7620-140194058984448\" (UID: \"f4ce1fc4-058d-4782-8c43-3df376b68a34\") " pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:01.057212 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.057121 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96bj7\" (UniqueName: \"kubernetes.io/projected/f4ce1fc4-058d-4782-8c43-3df376b68a34-kube-api-access-96bj7\") pod \"test-curl-7620-140194058984448\" (UID: \"f4ce1fc4-058d-4782-8c43-3df376b68a34\") " pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:01.066726 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.066697 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96bj7\" (UniqueName: \"kubernetes.io/projected/f4ce1fc4-058d-4782-8c43-3df376b68a34-kube-api-access-96bj7\") pod \"test-curl-7620-140194058984448\" (UID: \"f4ce1fc4-058d-4782-8c43-3df376b68a34\") " pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:01.100690 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.100648 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:01.251237 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.251210 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:53:01.252400 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:53:01.252371 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4ce1fc4_058d_4782_8c43_3df376b68a34.slice/crio-d1a1eef4e2c6992646a1a79c51a7cd4d056ed8aba35f4668b04bfbb0cc8897fe WatchSource:0}: Error finding container d1a1eef4e2c6992646a1a79c51a7cd4d056ed8aba35f4668b04bfbb0cc8897fe: Status 404 returned error can't find the container with id d1a1eef4e2c6992646a1a79c51a7cd4d056ed8aba35f4668b04bfbb0cc8897fe Jul 29 14:53:01.380995 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.380914 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cdd012d-254d-444d-8757-216ee33727c2" path="/var/lib/kubelet/pods/7cdd012d-254d-444d-8757-216ee33727c2/volumes" Jul 29 14:53:01.455110 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.455078 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:01.455333 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.455104 2560 scope.go:117] "RemoveContainer" containerID="523cdd136c6170257771b9bddd22f7a144278448750f231631f07e16b48c0d01" Jul 29 14:53:01.456610 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.456581 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194058984448" event={"ID":"f4ce1fc4-058d-4782-8c43-3df376b68a34","Type":"ContainerStarted","Data":"d1a1eef4e2c6992646a1a79c51a7cd4d056ed8aba35f4668b04bfbb0cc8897fe"} Jul 29 14:53:01.456723 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.456693 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" podUID="00542df7-5678-40c5-812d-50a32517898a" containerName="registry-server" containerID="cri-o://fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f" gracePeriod=2 Jul 29 14:53:01.740421 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.740373 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:53:01.866357 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.866319 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q66d\" (UniqueName: \"kubernetes.io/projected/00542df7-5678-40c5-812d-50a32517898a-kube-api-access-9q66d\") pod \"00542df7-5678-40c5-812d-50a32517898a\" (UID: \"00542df7-5678-40c5-812d-50a32517898a\") " Jul 29 14:53:01.868566 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.868524 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00542df7-5678-40c5-812d-50a32517898a-kube-api-access-9q66d" (OuterVolumeSpecName: "kube-api-access-9q66d") pod "00542df7-5678-40c5-812d-50a32517898a" (UID: "00542df7-5678-40c5-812d-50a32517898a"). InnerVolumeSpecName "kube-api-access-9q66d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:53:01.967341 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:01.967299 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q66d\" (UniqueName: \"kubernetes.io/projected/00542df7-5678-40c5-812d-50a32517898a-kube-api-access-9q66d\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:53:02.465444 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.465409 2560 generic.go:356] "Generic (PLEG): container finished" podID="00542df7-5678-40c5-812d-50a32517898a" containerID="fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f" exitCode=0 Jul 29 14:53:02.465642 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.465457 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" Jul 29 14:53:02.465642 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.465491 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" event={"ID":"00542df7-5678-40c5-812d-50a32517898a","Type":"ContainerDied","Data":"fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f"} Jul 29 14:53:02.465642 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.465577 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n5hbk" event={"ID":"00542df7-5678-40c5-812d-50a32517898a","Type":"ContainerDied","Data":"3fd6b47b46dbdd983be1d2afba64f41965e358ac9687737095dfaf51b62aa39d"} Jul 29 14:53:02.465642 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.465606 2560 scope.go:117] "RemoveContainer" containerID="fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f" Jul 29 14:53:02.467177 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.467155 2560 generic.go:356] "Generic (PLEG): container finished" podID="f4ce1fc4-058d-4782-8c43-3df376b68a34" containerID="ec650895f8a2c9b60356fbd4fb6f73dadc4598abc416ad990e07789b8dd10eef" exitCode=0 Jul 29 14:53:02.467271 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.467252 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7620-140194058984448" event={"ID":"f4ce1fc4-058d-4782-8c43-3df376b68a34","Type":"ContainerDied","Data":"ec650895f8a2c9b60356fbd4fb6f73dadc4598abc416ad990e07789b8dd10eef"} Jul 29 14:53:02.476994 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.476973 2560 scope.go:117] "RemoveContainer" containerID="fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f" Jul 29 14:53:02.477331 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:53:02.477295 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f\": container with ID starting with fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f not found: ID does not exist" containerID="fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f" Jul 29 14:53:02.477398 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.477338 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f"} err="failed to get container status \"fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f\": rpc error: code = NotFound desc = could not find container \"fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f\": container with ID starting with fea9824d515a5f62b6e09718d325d16148b1689d99db7a3456ebeb9798de9e0f not found: ID does not exist" Jul 29 14:53:02.499582 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.499524 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n5hbk"] Jul 29 14:53:02.502375 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:02.502349 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n5hbk"] Jul 29 14:53:03.382029 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:03.381986 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00542df7-5678-40c5-812d-50a32517898a" path="/var/lib/kubelet/pods/00542df7-5678-40c5-812d-50a32517898a/volumes" Jul 29 14:53:03.641440 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:03.641371 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:03.662985 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:03.662955 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7620-140194058984448_f4ce1fc4-058d-4782-8c43-3df376b68a34/test-curl-7620-140194058984448/0.log" Jul 29 14:53:03.688669 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:03.688633 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:53:03.691335 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:03.691303 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7620-140194058984448"] Jul 29 14:53:03.784931 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:03.784898 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96bj7\" (UniqueName: \"kubernetes.io/projected/f4ce1fc4-058d-4782-8c43-3df376b68a34-kube-api-access-96bj7\") pod \"f4ce1fc4-058d-4782-8c43-3df376b68a34\" (UID: \"f4ce1fc4-058d-4782-8c43-3df376b68a34\") " Jul 29 14:53:03.787185 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:03.787148 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ce1fc4-058d-4782-8c43-3df376b68a34-kube-api-access-96bj7" (OuterVolumeSpecName: "kube-api-access-96bj7") pod "f4ce1fc4-058d-4782-8c43-3df376b68a34" (UID: "f4ce1fc4-058d-4782-8c43-3df376b68a34"). InnerVolumeSpecName "kube-api-access-96bj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:53:03.886198 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:03.886142 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96bj7\" (UniqueName: \"kubernetes.io/projected/f4ce1fc4-058d-4782-8c43-3df376b68a34-kube-api-access-96bj7\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:53:04.483000 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:04.482971 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:04.483430 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:04.483016 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1a1eef4e2c6992646a1a79c51a7cd4d056ed8aba35f4668b04bfbb0cc8897fe" Jul 29 14:53:04.495486 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:53:04.495445 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7620-140194058984448\" is forbidden: User \"system:node:ip-10-0-139-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-155.ec2.internal' and this object" podUID="f4ce1fc4-058d-4782-8c43-3df376b68a34" pod="opendatahub/test-curl-7620-140194058984448" Jul 29 14:53:05.382101 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:05.382057 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ce1fc4-058d-4782-8c43-3df376b68a34" path="/var/lib/kubelet/pods/f4ce1fc4-058d-4782-8c43-3df376b68a34/volumes" Jul 29 14:53:08.212719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.212672 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb"] Jul 29 14:53:08.213298 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.213279 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00542df7-5678-40c5-812d-50a32517898a" containerName="registry-server" Jul 29 14:53:08.213298 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.213301 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="00542df7-5678-40c5-812d-50a32517898a" containerName="registry-server" Jul 29 14:53:08.213387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.213321 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ce1fc4-058d-4782-8c43-3df376b68a34" containerName="test-curl-7620-140194058984448" Jul 29 14:53:08.213387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.213327 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ce1fc4-058d-4782-8c43-3df376b68a34" containerName="test-curl-7620-140194058984448" Jul 29 14:53:08.213465 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.213395 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="00542df7-5678-40c5-812d-50a32517898a" containerName="registry-server" Jul 29 14:53:08.213465 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.213402 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ce1fc4-058d-4782-8c43-3df376b68a34" containerName="test-curl-7620-140194058984448" Jul 29 14:53:08.218051 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.218033 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.221556 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.221518 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-19afc1-serving-cert\"" Jul 29 14:53:08.232678 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.232646 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb"] Jul 29 14:53:08.329068 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.328968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47xr\" (UniqueName: \"kubernetes.io/projected/66c8252b-5db3-43b6-827e-692b2b702c87-kube-api-access-x47xr\") pod \"maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.329068 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.329048 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c8252b-5db3-43b6-827e-692b2b702c87-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.329273 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.329136 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c8252b-5db3-43b6-827e-692b2b702c87-maas-api-tls\") pod \"maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.430761 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.430723 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47xr\" (UniqueName: \"kubernetes.io/projected/66c8252b-5db3-43b6-827e-692b2b702c87-kube-api-access-x47xr\") pod \"maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.430954 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.430809 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c8252b-5db3-43b6-827e-692b2b702c87-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.430954 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.430852 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c8252b-5db3-43b6-827e-692b2b702c87-maas-api-tls\") pod \"maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.431634 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.431613 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c8252b-5db3-43b6-827e-692b2b702c87-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.433666 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.433646 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c8252b-5db3-43b6-827e-692b2b702c87-maas-api-tls\") pod \"maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.444782 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.444752 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47xr\" (UniqueName: \"kubernetes.io/projected/66c8252b-5db3-43b6-827e-692b2b702c87-kube-api-access-x47xr\") pod \"maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.530211 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.530171 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:08.694238 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:08.694208 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb"] Jul 29 14:53:08.696802 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:53:08.696772 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c8252b_5db3_43b6_827e_692b2b702c87.slice/crio-991dd9ad276d15b48676bdec8f34e432a9c8b5f373cf2cad88d2be831fa42532 WatchSource:0}: Error finding container 991dd9ad276d15b48676bdec8f34e432a9c8b5f373cf2cad88d2be831fa42532: Status 404 returned error can't find the container with id 991dd9ad276d15b48676bdec8f34e432a9c8b5f373cf2cad88d2be831fa42532 Jul 29 14:53:09.517927 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:09.517891 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" event={"ID":"66c8252b-5db3-43b6-827e-692b2b702c87","Type":"ContainerStarted","Data":"7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae"} Jul 29 14:53:09.518401 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:09.517944 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" event={"ID":"66c8252b-5db3-43b6-827e-692b2b702c87","Type":"ContainerStarted","Data":"991dd9ad276d15b48676bdec8f34e432a9c8b5f373cf2cad88d2be831fa42532"} Jul 29 14:53:09.518401 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:09.517977 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:09.538737 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:09.538676 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" podStartSLOduration=0.922380627 podStartE2EDuration="1.538660033s" totalImagesPullingTime="616.279406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:53:09.535339629 +0000 UTC m=+3122.736840803" watchObservedRunningTime="2026-07-29 14:53:09.538660033 +0000 UTC m=+3122.740161206" Jul 29 14:53:15.532308 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:15.532278 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:53:16.653463 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:16.653422 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:53:37.210672 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.210428 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8"] Jul 29 14:53:37.215068 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.215042 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.217570 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.217521 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-726aca-serving-cert\"" Jul 29 14:53:37.233720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.233687 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8"] Jul 29 14:53:37.322982 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.322946 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz6xk\" (UniqueName: \"kubernetes.io/projected/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-kube-api-access-mz6xk\") pod \"maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.323158 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.323025 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.323158 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.323083 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.424287 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.424251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.424474 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.424347 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz6xk\" (UniqueName: \"kubernetes.io/projected/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-kube-api-access-mz6xk\") pod \"maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.424474 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.424421 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.425083 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.425056 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.427221 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.427197 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.433742 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.433720 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz6xk\" (UniqueName: \"kubernetes.io/projected/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-kube-api-access-mz6xk\") pod \"maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.528381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.528287 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:37.693044 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:37.693014 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8"] Jul 29 14:53:37.694925 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:53:37.694856 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3c6bc2_e101_48b4_aec9_52c7b2f5ff32.slice/crio-76e236e57d42cbb454767629f281b3ad714d1ac1d4f2be104fd0ec5258bdd6bd WatchSource:0}: Error finding container 76e236e57d42cbb454767629f281b3ad714d1ac1d4f2be104fd0ec5258bdd6bd: Status 404 returned error can't find the container with id 76e236e57d42cbb454767629f281b3ad714d1ac1d4f2be104fd0ec5258bdd6bd Jul 29 14:53:38.709885 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:38.709844 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" event={"ID":"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32","Type":"ContainerStarted","Data":"bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048"} Jul 29 14:53:38.710355 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:38.709914 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:38.710355 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:38.709935 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" event={"ID":"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32","Type":"ContainerStarted","Data":"76e236e57d42cbb454767629f281b3ad714d1ac1d4f2be104fd0ec5258bdd6bd"} Jul 29 14:53:38.726711 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:38.726654 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" podStartSLOduration=1.401310428 podStartE2EDuration="1.726637592s" totalImagesPullingTime="325.327164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:53:38.725334727 +0000 UTC m=+3151.926835899" watchObservedRunningTime="2026-07-29 14:53:38.726637592 +0000 UTC m=+3151.928138764" Jul 29 14:53:44.722753 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:44.722720 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:53:46.954906 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:53:46.954872 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:54:15.550151 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:15.550116 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:54:44.048333 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:44.048292 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:54:54.759556 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.759507 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2"] Jul 29 14:54:54.763427 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.763409 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.766344 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.766319 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-ad0bed-openshift-default-dockercfg-dpn54\"" Jul 29 14:54:54.776873 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.776846 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2"] Jul 29 14:54:54.894341 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.894302 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-podinfo\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.894556 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.894348 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-socket\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.894556 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.894473 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-envoy\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.894717 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.894529 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-credential-socket\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.894717 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.894588 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-token\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.894717 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.894612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdhjp\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-kube-api-access-zdhjp\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.894717 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.894681 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-certs\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.894878 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.894716 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8089c8-385b-4fa6-a1b1-f66d33939061-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.894878 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.894812 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-data\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.996386 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996351 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-data\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.996619 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996450 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-podinfo\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.996619 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-socket\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.996619 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996522 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-envoy\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.996619 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-credential-socket\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.996858 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-token\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.996858 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996657 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhjp\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-kube-api-access-zdhjp\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.996858 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-certs\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.996858 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996736 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8089c8-385b-4fa6-a1b1-f66d33939061-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.997072 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996866 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-data\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.997072 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.996999 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-credential-socket\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.997218 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.997095 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-certs\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.997343 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.997323 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-socket\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.997674 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.997649 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8089c8-385b-4fa6-a1b1-f66d33939061-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.999141 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.999121 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-envoy\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:54.999433 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:54.999415 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-podinfo\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:55.005807 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:55.005779 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-token\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:55.005917 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:55.005899 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhjp\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-kube-api-access-zdhjp\") pod \"e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:55.075178 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:55.075082 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:55.236906 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:55.236878 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2"] Jul 29 14:54:55.238255 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:54:55.238226 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8089c8_385b_4fa6_a1b1_f66d33939061.slice/crio-f44ed354222e26392b4dbaffba1f6f444fbdee14415757e833070b9f97eb5104 WatchSource:0}: Error finding container f44ed354222e26392b4dbaffba1f6f444fbdee14415757e833070b9f97eb5104: Status 404 returned error can't find the container with id f44ed354222e26392b4dbaffba1f6f444fbdee14415757e833070b9f97eb5104 Jul 29 14:54:55.240861 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:55.240807 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:54:55.240950 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:55.240916 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:54:55.240989 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:55.240968 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:54:56.226965 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:56.226925 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" event={"ID":"7d8089c8-385b-4fa6-a1b1-f66d33939061","Type":"ContainerStarted","Data":"d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9"} Jul 29 14:54:56.227388 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:56.226990 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" event={"ID":"7d8089c8-385b-4fa6-a1b1-f66d33939061","Type":"ContainerStarted","Data":"f44ed354222e26392b4dbaffba1f6f444fbdee14415757e833070b9f97eb5104"} Jul 29 14:54:56.250377 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:56.250314 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" podStartSLOduration=2.25029535 podStartE2EDuration="2.25029535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:54:56.248004917 +0000 UTC m=+3229.449506089" watchObservedRunningTime="2026-07-29 14:54:56.25029535 +0000 UTC m=+3229.451796522" Jul 29 14:54:57.075870 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:57.075832 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:57.081239 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:57.081205 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:57.233860 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:57.233827 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:57.234720 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:57.234703 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:54:58.394226 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.394132 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5"] Jul 29 14:54:58.399107 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.399080 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:58.404815 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.404787 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ad0bed-serving-cert\"" Jul 29 14:54:58.407428 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.407397 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5"] Jul 29 14:54:58.533719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.533684 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxpp5\" (UniqueName: \"kubernetes.io/projected/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-kube-api-access-gxpp5\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:58.533878 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.533757 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:58.533946 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.533895 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:58.634716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.634675 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:58.634919 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.634745 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxpp5\" (UniqueName: \"kubernetes.io/projected/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-kube-api-access-gxpp5\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:58.634919 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.634788 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:58.634919 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:54:58.634904 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-serving-cert: secret "maas-api-e2e-ipp-cleanup-ad0bed-serving-cert" not found Jul 29 14:54:58.635054 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:54:58.634959 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-maas-api-tls podName:24509dc9-a00d-4fbb-9abd-a7fad395b9a7 nodeName:}" failed. No retries permitted until 2026-07-29 14:54:59.134937575 +0000 UTC m=+3232.336438724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" (UID: "24509dc9-a00d-4fbb-9abd-a7fad395b9a7") : secret "maas-api-e2e-ipp-cleanup-ad0bed-serving-cert" not found Jul 29 14:54:58.635380 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.635360 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:58.643771 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:58.643740 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxpp5\" (UniqueName: \"kubernetes.io/projected/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-kube-api-access-gxpp5\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:59.140557 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:59.140489 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:59.143174 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:59.143143 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:59.312640 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:59.312602 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:54:59.523614 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:54:59.523583 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5"] Jul 29 14:54:59.524766 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:54:59.524733 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24509dc9_a00d_4fbb_9abd_a7fad395b9a7.slice/crio-b1150c22573d44357b9de433de49a4f13ceee0ff06f34eee048e37e828f820e9 WatchSource:0}: Error finding container b1150c22573d44357b9de433de49a4f13ceee0ff06f34eee048e37e828f820e9: Status 404 returned error can't find the container with id b1150c22573d44357b9de433de49a4f13ceee0ff06f34eee048e37e828f820e9 Jul 29 14:55:00.255614 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:00.255576 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" event={"ID":"24509dc9-a00d-4fbb-9abd-a7fad395b9a7","Type":"ContainerStarted","Data":"f3936aaf1c4b38e1ac3a9a05c20ed1e7cf251c1398c1a0ab179c619cee2c1548"} Jul 29 14:55:00.255811 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:00.255641 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" event={"ID":"24509dc9-a00d-4fbb-9abd-a7fad395b9a7","Type":"ContainerStarted","Data":"b1150c22573d44357b9de433de49a4f13ceee0ff06f34eee048e37e828f820e9"} Jul 29 14:55:00.255811 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:00.255802 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:55:00.276009 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:00.275954 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" podStartSLOduration=1.769766432 podStartE2EDuration="2.275938203s" totalImagesPullingTime="506.171771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:55:00.272766454 +0000 UTC m=+3233.474267626" watchObservedRunningTime="2026-07-29 14:55:00.275938203 +0000 UTC m=+3233.477439423" Jul 29 14:55:06.267331 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:06.267299 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:55:08.559610 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:08.559572 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:55:17.386348 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:17.386304 2560 generic.go:356] "Generic (PLEG): container finished" podID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerID="d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9" exitCode=0 Jul 29 14:55:17.389571 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:17.389526 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" event={"ID":"7d8089c8-385b-4fa6-a1b1-f66d33939061","Type":"ContainerDied","Data":"d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9"} Jul 29 14:55:17.389980 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:17.389962 2560 scope.go:117] "RemoveContainer" containerID="d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9" Jul 29 14:55:17.391396 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:17.391358 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:55:17.391475 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:17.391457 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:55:17.391531 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:17.391514 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Jul 29 14:55:18.399325 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:18.399275 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" event={"ID":"7d8089c8-385b-4fa6-a1b1-f66d33939061","Type":"ContainerStarted","Data":"59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c"} Jul 29 14:55:19.076253 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:19.076210 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:55:19.076454 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:19.076434 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Jul 29 14:55:19.076509 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:19.076487 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Jul 29 14:55:20.076432 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:20.076395 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Jul 29 14:55:20.076881 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:20.076452 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Jul 29 14:55:21.075737 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:21.075692 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Jul 29 14:55:21.075934 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:21.075771 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Jul 29 14:55:22.076224 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:22.076190 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Jul 29 14:55:22.076681 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:22.076248 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Jul 29 14:55:23.079868 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:23.079839 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:55:23.080362 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:23.080337 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:55:23.081133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:23.081114 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:55:23.908994 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:23.908955 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8"] Jul 29 14:55:23.912636 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:23.912618 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" Jul 29 14:55:23.915632 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:23.915608 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:55:23.921711 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:23.921165 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8"] Jul 29 14:55:24.012656 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:24.012617 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d15be1-21b4-45f9-9411-46622de93fea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8\" (UID: \"d4d15be1-21b4-45f9-9411-46622de93fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" Jul 29 14:55:24.114111 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:24.114064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d15be1-21b4-45f9-9411-46622de93fea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8\" (UID: \"d4d15be1-21b4-45f9-9411-46622de93fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" Jul 29 14:55:24.114762 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:24.114743 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d15be1-21b4-45f9-9411-46622de93fea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8\" (UID: \"d4d15be1-21b4-45f9-9411-46622de93fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" Jul 29 14:55:24.226415 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:24.226367 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" Jul 29 14:55:24.380320 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:24.380290 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8"] Jul 29 14:55:24.381715 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:55:24.381684 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d15be1_21b4_45f9_9411_46622de93fea.slice/crio-3af04cd685da68c606a23a672500bd44efc26fad621ddfc68eb4c9904917a753 WatchSource:0}: Error finding container 3af04cd685da68c606a23a672500bd44efc26fad621ddfc68eb4c9904917a753: Status 404 returned error can't find the container with id 3af04cd685da68c606a23a672500bd44efc26fad621ddfc68eb4c9904917a753 Jul 29 14:55:24.446979 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:24.446946 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" event={"ID":"d4d15be1-21b4-45f9-9411-46622de93fea","Type":"ContainerStarted","Data":"3af04cd685da68c606a23a672500bd44efc26fad621ddfc68eb4c9904917a753"} Jul 29 14:55:25.455154 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:25.455120 2560 generic.go:356] "Generic (PLEG): container finished" podID="d4d15be1-21b4-45f9-9411-46622de93fea" containerID="a60cd4b745ad698bb5609f8bb7c478f6f000db1ae12068965dce55f0ce9384b6" exitCode=0 Jul 29 14:55:25.455606 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:25.455217 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" event={"ID":"d4d15be1-21b4-45f9-9411-46622de93fea","Type":"ContainerDied","Data":"a60cd4b745ad698bb5609f8bb7c478f6f000db1ae12068965dce55f0ce9384b6"} Jul 29 14:55:26.610718 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:26.610689 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" Jul 29 14:55:26.641342 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:26.641307 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d15be1-21b4-45f9-9411-46622de93fea-maas-api-service-ca\") pod \"d4d15be1-21b4-45f9-9411-46622de93fea\" (UID: \"d4d15be1-21b4-45f9-9411-46622de93fea\") " Jul 29 14:55:26.641813 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:26.641786 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d15be1-21b4-45f9-9411-46622de93fea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4d15be1-21b4-45f9-9411-46622de93fea" (UID: "d4d15be1-21b4-45f9-9411-46622de93fea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:55:26.742506 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:26.742409 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d15be1-21b4-45f9-9411-46622de93fea-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:27.469731 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:27.469703 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" Jul 29 14:55:27.469933 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:27.469734 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8" event={"ID":"d4d15be1-21b4-45f9-9411-46622de93fea","Type":"ContainerDied","Data":"3af04cd685da68c606a23a672500bd44efc26fad621ddfc68eb4c9904917a753"} Jul 29 14:55:27.469933 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:27.469781 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af04cd685da68c606a23a672500bd44efc26fad621ddfc68eb4c9904917a753" Jul 29 14:55:29.293248 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.293207 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5"] Jul 29 14:55:29.293704 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.293426 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" podUID="24509dc9-a00d-4fbb-9abd-a7fad395b9a7" containerName="maas-api" containerID="cri-o://f3936aaf1c4b38e1ac3a9a05c20ed1e7cf251c1398c1a0ab179c619cee2c1548" gracePeriod=30 Jul 29 14:55:29.485924 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.485894 2560 generic.go:356] "Generic (PLEG): container finished" podID="24509dc9-a00d-4fbb-9abd-a7fad395b9a7" containerID="f3936aaf1c4b38e1ac3a9a05c20ed1e7cf251c1398c1a0ab179c619cee2c1548" exitCode=0 Jul 29 14:55:29.486083 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.485969 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" event={"ID":"24509dc9-a00d-4fbb-9abd-a7fad395b9a7","Type":"ContainerDied","Data":"f3936aaf1c4b38e1ac3a9a05c20ed1e7cf251c1398c1a0ab179c619cee2c1548"} Jul 29 14:55:29.573225 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.573196 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:55:29.674097 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.674059 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-maas-api-tls\") pod \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " Jul 29 14:55:29.674097 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.674102 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-openshift-service-ca\") pod \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " Jul 29 14:55:29.674379 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.674133 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxpp5\" (UniqueName: \"kubernetes.io/projected/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-kube-api-access-gxpp5\") pod \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\" (UID: \"24509dc9-a00d-4fbb-9abd-a7fad395b9a7\") " Jul 29 14:55:29.674515 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.674488 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24509dc9-a00d-4fbb-9abd-a7fad395b9a7" (UID: "24509dc9-a00d-4fbb-9abd-a7fad395b9a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:55:29.676656 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.676620 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-kube-api-access-gxpp5" (OuterVolumeSpecName: "kube-api-access-gxpp5") pod "24509dc9-a00d-4fbb-9abd-a7fad395b9a7" (UID: "24509dc9-a00d-4fbb-9abd-a7fad395b9a7"). InnerVolumeSpecName "kube-api-access-gxpp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:55:29.677102 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.677021 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24509dc9-a00d-4fbb-9abd-a7fad395b9a7" (UID: "24509dc9-a00d-4fbb-9abd-a7fad395b9a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:55:29.776142 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.776102 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-maas-api-tls\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:29.776142 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.776143 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-openshift-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:29.776356 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.776162 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxpp5\" (UniqueName: \"kubernetes.io/projected/24509dc9-a00d-4fbb-9abd-a7fad395b9a7-kube-api-access-gxpp5\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:29.780935 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.780900 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8"] Jul 29 14:55:29.784010 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:29.783983 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad0bed-8t9l8"] Jul 29 14:55:30.495105 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:30.495072 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" Jul 29 14:55:30.495572 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:30.495081 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5" event={"ID":"24509dc9-a00d-4fbb-9abd-a7fad395b9a7","Type":"ContainerDied","Data":"b1150c22573d44357b9de433de49a4f13ceee0ff06f34eee048e37e828f820e9"} Jul 29 14:55:30.495572 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:30.495204 2560 scope.go:117] "RemoveContainer" containerID="f3936aaf1c4b38e1ac3a9a05c20ed1e7cf251c1398c1a0ab179c619cee2c1548" Jul 29 14:55:30.520979 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:30.520937 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5"] Jul 29 14:55:30.524147 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:30.524109 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad0bed-67449b945d-rrqx5"] Jul 29 14:55:31.382194 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:31.382156 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24509dc9-a00d-4fbb-9abd-a7fad395b9a7" path="/var/lib/kubelet/pods/24509dc9-a00d-4fbb-9abd-a7fad395b9a7/volumes" Jul 29 14:55:31.382592 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:31.382577 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4d15be1-21b4-45f9-9411-46622de93fea" path="/var/lib/kubelet/pods/d4d15be1-21b4-45f9-9411-46622de93fea/volumes" Jul 29 14:55:33.806872 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:33.806828 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:55:37.256921 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:37.256878 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2"] Jul 29 14:55:37.257408 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:37.257107 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" containerID="cri-o://59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c" gracePeriod=30 Jul 29 14:55:38.083196 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:38.083149 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 14:55:38.083389 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:38.083237 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 14:55:41.230930 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.230889 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6"] Jul 29 14:55:41.231393 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.231381 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24509dc9-a00d-4fbb-9abd-a7fad395b9a7" containerName="maas-api" Jul 29 14:55:41.231438 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.231395 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="24509dc9-a00d-4fbb-9abd-a7fad395b9a7" containerName="maas-api" Jul 29 14:55:41.231438 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.231407 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d15be1-21b4-45f9-9411-46622de93fea" containerName="revoke-keys" Jul 29 14:55:41.231438 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.231412 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d15be1-21b4-45f9-9411-46622de93fea" containerName="revoke-keys" Jul 29 14:55:41.231575 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.231490 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="24509dc9-a00d-4fbb-9abd-a7fad395b9a7" containerName="maas-api" Jul 29 14:55:41.231575 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.231499 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d15be1-21b4-45f9-9411-46622de93fea" containerName="revoke-keys" Jul 29 14:55:41.236491 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.236470 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" Jul 29 14:55:41.239378 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.239357 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:55:41.246375 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.246348 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6"] Jul 29 14:55:41.399628 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.399583 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4e242-8bf2-431e-9aad-9bb715be5a29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6\" (UID: \"eac4e242-8bf2-431e-9aad-9bb715be5a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" Jul 29 14:55:41.500489 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.500381 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4e242-8bf2-431e-9aad-9bb715be5a29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6\" (UID: \"eac4e242-8bf2-431e-9aad-9bb715be5a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" Jul 29 14:55:41.501074 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.501051 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4e242-8bf2-431e-9aad-9bb715be5a29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6\" (UID: \"eac4e242-8bf2-431e-9aad-9bb715be5a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" Jul 29 14:55:41.549907 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.549866 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" Jul 29 14:55:41.715859 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:41.715832 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6"] Jul 29 14:55:41.717807 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:55:41.717781 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac4e242_8bf2_431e_9aad_9bb715be5a29.slice/crio-3a0dea5b49ab620e0ad4fe811af8c5c3c38ec2a109400c3bd30e9895f02a3e49 WatchSource:0}: Error finding container 3a0dea5b49ab620e0ad4fe811af8c5c3c38ec2a109400c3bd30e9895f02a3e49: Status 404 returned error can't find the container with id 3a0dea5b49ab620e0ad4fe811af8c5c3c38ec2a109400c3bd30e9895f02a3e49 Jul 29 14:55:42.514184 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.514159 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:55:42.583723 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.583685 2560 generic.go:356] "Generic (PLEG): container finished" podID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerID="59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c" exitCode=0 Jul 29 14:55:42.583916 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.583749 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" Jul 29 14:55:42.583916 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.583774 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" event={"ID":"7d8089c8-385b-4fa6-a1b1-f66d33939061","Type":"ContainerDied","Data":"59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c"} Jul 29 14:55:42.583916 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.583831 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2" event={"ID":"7d8089c8-385b-4fa6-a1b1-f66d33939061","Type":"ContainerDied","Data":"f44ed354222e26392b4dbaffba1f6f444fbdee14415757e833070b9f97eb5104"} Jul 29 14:55:42.583916 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.583861 2560 scope.go:117] "RemoveContainer" containerID="59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c" Jul 29 14:55:42.585439 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.585417 2560 generic.go:356] "Generic (PLEG): container finished" podID="eac4e242-8bf2-431e-9aad-9bb715be5a29" containerID="f6e6489c09ad535d1136bf8e101aefb63fc07ab3697ac9962fa417a846575eaa" exitCode=0 Jul 29 14:55:42.585584 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.585504 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" event={"ID":"eac4e242-8bf2-431e-9aad-9bb715be5a29","Type":"ContainerDied","Data":"f6e6489c09ad535d1136bf8e101aefb63fc07ab3697ac9962fa417a846575eaa"} Jul 29 14:55:42.585584 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.585578 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" event={"ID":"eac4e242-8bf2-431e-9aad-9bb715be5a29","Type":"ContainerStarted","Data":"3a0dea5b49ab620e0ad4fe811af8c5c3c38ec2a109400c3bd30e9895f02a3e49"} Jul 29 14:55:42.596986 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.596963 2560 scope.go:117] "RemoveContainer" containerID="d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9" Jul 29 14:55:42.607367 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.607343 2560 scope.go:117] "RemoveContainer" containerID="59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c" Jul 29 14:55:42.607769 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:55:42.607748 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c\": container with ID starting with 59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c not found: ID does not exist" containerID="59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c" Jul 29 14:55:42.607843 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.607778 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c"} err="failed to get container status \"59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c\": rpc error: code = NotFound desc = could not find container \"59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c\": container with ID starting with 59252668d44fc23d3e5864e598b8853c0c90beaa8aa05d471e111a791038f04c not found: ID does not exist" Jul 29 14:55:42.607843 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.607798 2560 scope.go:117] "RemoveContainer" containerID="d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9" Jul 29 14:55:42.608046 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:55:42.608028 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9\": container with ID starting with d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9 not found: ID does not exist" containerID="d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9" Jul 29 14:55:42.608097 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.608049 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9"} err="failed to get container status \"d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9\": rpc error: code = NotFound desc = could not find container \"d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9\": container with ID starting with d4a3583d93a4b6dc5f51fbda8576c265683987dc4814a13be36d809bb50d36b9 not found: ID does not exist" Jul 29 14:55:42.611375 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611355 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-credential-socket\") pod \"7d8089c8-385b-4fa6-a1b1-f66d33939061\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " Jul 29 14:55:42.611474 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611394 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-token\") pod \"7d8089c8-385b-4fa6-a1b1-f66d33939061\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " Jul 29 14:55:42.611474 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611460 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8089c8-385b-4fa6-a1b1-f66d33939061-istiod-ca-cert\") pod \"7d8089c8-385b-4fa6-a1b1-f66d33939061\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " Jul 29 14:55:42.611607 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611491 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-data\") pod \"7d8089c8-385b-4fa6-a1b1-f66d33939061\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " Jul 29 14:55:42.611607 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611518 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-envoy\") pod \"7d8089c8-385b-4fa6-a1b1-f66d33939061\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " Jul 29 14:55:42.611607 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611575 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-socket\") pod \"7d8089c8-385b-4fa6-a1b1-f66d33939061\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " Jul 29 14:55:42.611607 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611602 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdhjp\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-kube-api-access-zdhjp\") pod \"7d8089c8-385b-4fa6-a1b1-f66d33939061\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " Jul 29 14:55:42.611815 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611647 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d8089c8-385b-4fa6-a1b1-f66d33939061" (UID: "7d8089c8-385b-4fa6-a1b1-f66d33939061"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:55:42.611815 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611682 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-certs\") pod \"7d8089c8-385b-4fa6-a1b1-f66d33939061\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " Jul 29 14:55:42.611815 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611765 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-podinfo\") pod \"7d8089c8-385b-4fa6-a1b1-f66d33939061\" (UID: \"7d8089c8-385b-4fa6-a1b1-f66d33939061\") " Jul 29 14:55:42.611962 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611891 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8089c8-385b-4fa6-a1b1-f66d33939061-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d8089c8-385b-4fa6-a1b1-f66d33939061" (UID: "7d8089c8-385b-4fa6-a1b1-f66d33939061"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:55:42.612021 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.611993 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d8089c8-385b-4fa6-a1b1-f66d33939061" (UID: "7d8089c8-385b-4fa6-a1b1-f66d33939061"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:55:42.612080 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.612041 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d8089c8-385b-4fa6-a1b1-f66d33939061" (UID: "7d8089c8-385b-4fa6-a1b1-f66d33939061"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:55:42.612260 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.612241 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-credential-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:42.612346 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.612267 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8089c8-385b-4fa6-a1b1-f66d33939061-istiod-ca-cert\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:42.612346 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.612281 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:42.612346 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.612293 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-workload-certs\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:42.612620 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.612593 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d8089c8-385b-4fa6-a1b1-f66d33939061" (UID: "7d8089c8-385b-4fa6-a1b1-f66d33939061"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:55:42.613885 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.613863 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d8089c8-385b-4fa6-a1b1-f66d33939061" (UID: "7d8089c8-385b-4fa6-a1b1-f66d33939061"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:55:42.614083 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.614061 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d8089c8-385b-4fa6-a1b1-f66d33939061" (UID: "7d8089c8-385b-4fa6-a1b1-f66d33939061"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 14:55:42.614422 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.614399 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-kube-api-access-zdhjp" (OuterVolumeSpecName: "kube-api-access-zdhjp") pod "7d8089c8-385b-4fa6-a1b1-f66d33939061" (UID: "7d8089c8-385b-4fa6-a1b1-f66d33939061"). InnerVolumeSpecName "kube-api-access-zdhjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:55:42.614422 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.614397 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d8089c8-385b-4fa6-a1b1-f66d33939061" (UID: "7d8089c8-385b-4fa6-a1b1-f66d33939061"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:55:42.713143 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.713104 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-podinfo\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:42.713143 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.713134 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-token\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:42.713143 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.713143 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-data\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:42.713143 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.713151 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8089c8-385b-4fa6-a1b1-f66d33939061-istio-envoy\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:42.713422 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.713159 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdhjp\" (UniqueName: \"kubernetes.io/projected/7d8089c8-385b-4fa6-a1b1-f66d33939061-kube-api-access-zdhjp\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:42.914924 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.914883 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2"] Jul 29 14:55:42.917647 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:42.917617 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ad0bed-openshift-default-866474cb6d-x4dv2"] Jul 29 14:55:43.381281 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:43.381245 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" path="/var/lib/kubelet/pods/7d8089c8-385b-4fa6-a1b1-f66d33939061/volumes" Jul 29 14:55:43.746430 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:43.746403 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" Jul 29 14:55:43.924947 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:43.924853 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4e242-8bf2-431e-9aad-9bb715be5a29-maas-api-service-ca\") pod \"eac4e242-8bf2-431e-9aad-9bb715be5a29\" (UID: \"eac4e242-8bf2-431e-9aad-9bb715be5a29\") " Jul 29 14:55:43.925214 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:43.925191 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac4e242-8bf2-431e-9aad-9bb715be5a29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eac4e242-8bf2-431e-9aad-9bb715be5a29" (UID: "eac4e242-8bf2-431e-9aad-9bb715be5a29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:55:44.026716 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:44.026672 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4e242-8bf2-431e-9aad-9bb715be5a29-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:44.602576 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:44.602530 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" Jul 29 14:55:44.602763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:44.602572 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6" event={"ID":"eac4e242-8bf2-431e-9aad-9bb715be5a29","Type":"ContainerDied","Data":"3a0dea5b49ab620e0ad4fe811af8c5c3c38ec2a109400c3bd30e9895f02a3e49"} Jul 29 14:55:44.602763 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:44.602620 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a0dea5b49ab620e0ad4fe811af8c5c3c38ec2a109400c3bd30e9895f02a3e49" Jul 29 14:55:56.580207 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:56.580166 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:55:57.676329 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:57.676291 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb"] Jul 29 14:55:57.676785 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:57.676523 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" podUID="66c8252b-5db3-43b6-827e-692b2b702c87" containerName="maas-api" containerID="cri-o://7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae" gracePeriod=30 Jul 29 14:55:57.958648 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:57.958621 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:55:58.057517 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.057479 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c8252b-5db3-43b6-827e-692b2b702c87-maas-api-tls\") pod \"66c8252b-5db3-43b6-827e-692b2b702c87\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " Jul 29 14:55:58.057718 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.057581 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c8252b-5db3-43b6-827e-692b2b702c87-openshift-service-ca\") pod \"66c8252b-5db3-43b6-827e-692b2b702c87\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " Jul 29 14:55:58.057718 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.057624 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x47xr\" (UniqueName: \"kubernetes.io/projected/66c8252b-5db3-43b6-827e-692b2b702c87-kube-api-access-x47xr\") pod \"66c8252b-5db3-43b6-827e-692b2b702c87\" (UID: \"66c8252b-5db3-43b6-827e-692b2b702c87\") " Jul 29 14:55:58.057941 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.057920 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c8252b-5db3-43b6-827e-692b2b702c87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66c8252b-5db3-43b6-827e-692b2b702c87" (UID: "66c8252b-5db3-43b6-827e-692b2b702c87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:55:58.058061 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.058043 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c8252b-5db3-43b6-827e-692b2b702c87-openshift-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:58.059992 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.059964 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c8252b-5db3-43b6-827e-692b2b702c87-kube-api-access-x47xr" (OuterVolumeSpecName: "kube-api-access-x47xr") pod "66c8252b-5db3-43b6-827e-692b2b702c87" (UID: "66c8252b-5db3-43b6-827e-692b2b702c87"). InnerVolumeSpecName "kube-api-access-x47xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:55:58.060106 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.060038 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c8252b-5db3-43b6-827e-692b2b702c87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66c8252b-5db3-43b6-827e-692b2b702c87" (UID: "66c8252b-5db3-43b6-827e-692b2b702c87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:55:58.088068 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.088029 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6"] Jul 29 14:55:58.094760 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.094718 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19afc1-mqcj6"] Jul 29 14:55:58.159450 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.159414 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47xr\" (UniqueName: \"kubernetes.io/projected/66c8252b-5db3-43b6-827e-692b2b702c87-kube-api-access-x47xr\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:58.159450 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.159443 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c8252b-5db3-43b6-827e-692b2b702c87-maas-api-tls\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:55:58.699466 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.699426 2560 generic.go:356] "Generic (PLEG): container finished" podID="66c8252b-5db3-43b6-827e-692b2b702c87" containerID="7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae" exitCode=0 Jul 29 14:55:58.699956 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.699476 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" Jul 29 14:55:58.699956 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.699520 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" event={"ID":"66c8252b-5db3-43b6-827e-692b2b702c87","Type":"ContainerDied","Data":"7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae"} Jul 29 14:55:58.699956 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.699602 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb" event={"ID":"66c8252b-5db3-43b6-827e-692b2b702c87","Type":"ContainerDied","Data":"991dd9ad276d15b48676bdec8f34e432a9c8b5f373cf2cad88d2be831fa42532"} Jul 29 14:55:58.699956 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.699630 2560 scope.go:117] "RemoveContainer" containerID="7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae" Jul 29 14:55:58.710424 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.710406 2560 scope.go:117] "RemoveContainer" containerID="7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae" Jul 29 14:55:58.710755 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:55:58.710733 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae\": container with ID starting with 7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae not found: ID does not exist" containerID="7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae" Jul 29 14:55:58.710814 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.710766 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae"} err="failed to get container status \"7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae\": rpc error: code = NotFound desc = could not find container \"7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae\": container with ID starting with 7933c35e1c31d4c9daa5a6fba37ce026ce6696bc2c95fc1bfecf9f13636548ae not found: ID does not exist" Jul 29 14:55:58.730977 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.730946 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb"] Jul 29 14:55:58.736474 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:58.736449 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19afc1-5b959ffddd-54mlb"] Jul 29 14:55:59.381560 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:59.381504 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c8252b-5db3-43b6-827e-692b2b702c87" path="/var/lib/kubelet/pods/66c8252b-5db3-43b6-827e-692b2b702c87/volumes" Jul 29 14:55:59.381922 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:55:59.381907 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac4e242-8bf2-431e-9aad-9bb715be5a29" path="/var/lib/kubelet/pods/eac4e242-8bf2-431e-9aad-9bb715be5a29/volumes" Jul 29 14:56:07.745225 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:07.745194 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:56:07.749158 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:07.749125 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:56:07.775138 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:07.775111 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:56:07.778689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:07.778668 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-plvns_721cdb29-0d68-4127-b176-0f7d020a53c4/ovn-acl-logging/0.log" Jul 29 14:56:08.162199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:08.162160 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:56:48.931033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.930948 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh"] Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931393 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931405 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931425 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac4e242-8bf2-431e-9aad-9bb715be5a29" containerName="revoke-keys" Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931430 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac4e242-8bf2-431e-9aad-9bb715be5a29" containerName="revoke-keys" Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931439 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c8252b-5db3-43b6-827e-692b2b702c87" containerName="maas-api" Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931445 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c8252b-5db3-43b6-827e-692b2b702c87" containerName="maas-api" Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931511 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac4e242-8bf2-431e-9aad-9bb715be5a29" containerName="revoke-keys" Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931521 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c8252b-5db3-43b6-827e-692b2b702c87" containerName="maas-api" Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931530 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" Jul 29 14:56:48.931527 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.931553 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" Jul 29 14:56:48.940259 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.940224 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" Jul 29 14:56:48.944181 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.944149 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:56:48.945106 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:48.945078 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh"] Jul 29 14:56:49.047028 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:49.046989 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e56878-35c9-4200-a903-7b703d2517cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh\" (UID: \"00e56878-35c9-4200-a903-7b703d2517cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" Jul 29 14:56:49.148661 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:49.148618 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e56878-35c9-4200-a903-7b703d2517cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh\" (UID: \"00e56878-35c9-4200-a903-7b703d2517cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" Jul 29 14:56:49.149302 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:49.149280 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e56878-35c9-4200-a903-7b703d2517cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh\" (UID: \"00e56878-35c9-4200-a903-7b703d2517cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" Jul 29 14:56:49.254927 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:49.254821 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" Jul 29 14:56:49.417961 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:49.417924 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh"] Jul 29 14:56:49.419419 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:56:49.419386 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e56878_35c9_4200_a903_7b703d2517cb.slice/crio-5430b9a3aa098d8eee08f31b497b9b8a7241f59f2462e2be8bef1aec5b16146f WatchSource:0}: Error finding container 5430b9a3aa098d8eee08f31b497b9b8a7241f59f2462e2be8bef1aec5b16146f: Status 404 returned error can't find the container with id 5430b9a3aa098d8eee08f31b497b9b8a7241f59f2462e2be8bef1aec5b16146f Jul 29 14:56:50.047068 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:50.047035 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" event={"ID":"00e56878-35c9-4200-a903-7b703d2517cb","Type":"ContainerStarted","Data":"6faaf06bc898ec7eed9bbfac8c59f5cdda7b4e4e6c33083b6298a2c1b4511c00"} Jul 29 14:56:50.047483 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:50.047082 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" event={"ID":"00e56878-35c9-4200-a903-7b703d2517cb","Type":"ContainerStarted","Data":"5430b9a3aa098d8eee08f31b497b9b8a7241f59f2462e2be8bef1aec5b16146f"} Jul 29 14:56:50.072384 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:50.072326 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" podStartSLOduration=2.072309174 podStartE2EDuration="2.072309174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:56:50.068986269 +0000 UTC m=+3343.270487440" watchObservedRunningTime="2026-07-29 14:56:50.072309174 +0000 UTC m=+3343.273810323" Jul 29 14:56:51.057227 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:51.057186 2560 generic.go:356] "Generic (PLEG): container finished" podID="00e56878-35c9-4200-a903-7b703d2517cb" containerID="6faaf06bc898ec7eed9bbfac8c59f5cdda7b4e4e6c33083b6298a2c1b4511c00" exitCode=0 Jul 29 14:56:51.057706 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:51.057267 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" event={"ID":"00e56878-35c9-4200-a903-7b703d2517cb","Type":"ContainerDied","Data":"6faaf06bc898ec7eed9bbfac8c59f5cdda7b4e4e6c33083b6298a2c1b4511c00"} Jul 29 14:56:52.209601 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:52.209577 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" Jul 29 14:56:52.277752 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:52.277713 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e56878-35c9-4200-a903-7b703d2517cb-maas-api-service-ca\") pod \"00e56878-35c9-4200-a903-7b703d2517cb\" (UID: \"00e56878-35c9-4200-a903-7b703d2517cb\") " Jul 29 14:56:52.278079 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:52.278055 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e56878-35c9-4200-a903-7b703d2517cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00e56878-35c9-4200-a903-7b703d2517cb" (UID: "00e56878-35c9-4200-a903-7b703d2517cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:56:52.378602 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:52.378484 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e56878-35c9-4200-a903-7b703d2517cb-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:56:53.071201 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:53.071167 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" event={"ID":"00e56878-35c9-4200-a903-7b703d2517cb","Type":"ContainerDied","Data":"5430b9a3aa098d8eee08f31b497b9b8a7241f59f2462e2be8bef1aec5b16146f"} Jul 29 14:56:53.071360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:53.071217 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5430b9a3aa098d8eee08f31b497b9b8a7241f59f2462e2be8bef1aec5b16146f" Jul 29 14:56:53.071360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:53.071180 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh" Jul 29 14:56:59.598097 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.598063 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8"] Jul 29 14:56:59.598663 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.598297 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" podUID="4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" containerName="maas-api" containerID="cri-o://bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048" gracePeriod=30 Jul 29 14:56:59.716840 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.716802 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" podUID="4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.91:8443/health\": dial tcp 10.132.0.91:8443: connect: connection refused" Jul 29 14:56:59.860281 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.860187 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:56:59.862112 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.862082 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh"] Jul 29 14:56:59.867646 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.867619 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-726aca-4x6wh"] Jul 29 14:56:59.951397 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.951361 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz6xk\" (UniqueName: \"kubernetes.io/projected/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-kube-api-access-mz6xk\") pod \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " Jul 29 14:56:59.951397 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.951414 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-openshift-service-ca\") pod \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " Jul 29 14:56:59.951689 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.951585 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-maas-api-tls\") pod \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\" (UID: \"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32\") " Jul 29 14:56:59.951883 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.951858 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" (UID: "4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:56:59.952003 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.951988 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-openshift-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:56:59.953887 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.953858 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" (UID: "4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:56:59.954007 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:56:59.953980 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-kube-api-access-mz6xk" (OuterVolumeSpecName: "kube-api-access-mz6xk") pod "4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" (UID: "4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32"). InnerVolumeSpecName "kube-api-access-mz6xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:00.052785 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.052740 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz6xk\" (UniqueName: \"kubernetes.io/projected/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-kube-api-access-mz6xk\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:00.052785 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.052777 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32-maas-api-tls\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:00.121855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.121758 2560 generic.go:356] "Generic (PLEG): container finished" podID="4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" containerID="bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048" exitCode=0 Jul 29 14:57:00.121855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.121810 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" Jul 29 14:57:00.121855 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.121840 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" event={"ID":"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32","Type":"ContainerDied","Data":"bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048"} Jul 29 14:57:00.122116 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.121893 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8" event={"ID":"4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32","Type":"ContainerDied","Data":"76e236e57d42cbb454767629f281b3ad714d1ac1d4f2be104fd0ec5258bdd6bd"} Jul 29 14:57:00.122116 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.121916 2560 scope.go:117] "RemoveContainer" containerID="bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048" Jul 29 14:57:00.134460 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.134431 2560 scope.go:117] "RemoveContainer" containerID="bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048" Jul 29 14:57:00.134937 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:57:00.134906 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048\": container with ID starting with bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048 not found: ID does not exist" containerID="bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048" Jul 29 14:57:00.135041 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.134953 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048"} err="failed to get container status \"bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048\": rpc error: code = NotFound desc = could not find container \"bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048\": container with ID starting with bc451ceecb91957a4b224b2dfe68df3effc86bf06fa62f83d43fe0cf6f5bf048 not found: ID does not exist" Jul 29 14:57:00.163927 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.163892 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8"] Jul 29 14:57:00.170739 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:00.170695 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-726aca-6c88f474db-nm7b8"] Jul 29 14:57:01.381734 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:01.381697 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e56878-35c9-4200-a903-7b703d2517cb" path="/var/lib/kubelet/pods/00e56878-35c9-4200-a903-7b703d2517cb/volumes" Jul 29 14:57:01.382164 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:01.382149 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" path="/var/lib/kubelet/pods/4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32/volumes" Jul 29 14:57:03.161754 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:03.161716 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:57:07.274176 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.274115 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf"] Jul 29 14:57:07.274670 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.274634 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e56878-35c9-4200-a903-7b703d2517cb" containerName="revoke-keys" Jul 29 14:57:07.274670 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.274649 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e56878-35c9-4200-a903-7b703d2517cb" containerName="revoke-keys" Jul 29 14:57:07.274670 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.274669 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" containerName="maas-api" Jul 29 14:57:07.274801 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.274675 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" containerName="maas-api" Jul 29 14:57:07.274801 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.274703 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" Jul 29 14:57:07.274801 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.274709 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8089c8-385b-4fa6-a1b1-f66d33939061" containerName="istio-proxy" Jul 29 14:57:07.274801 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.274786 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e3c6bc2-e101-48b4-aec9-52c7b2f5ff32" containerName="maas-api" Jul 29 14:57:07.274801 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.274794 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e56878-35c9-4200-a903-7b703d2517cb" containerName="revoke-keys" Jul 29 14:57:07.280104 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.280084 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" Jul 29 14:57:07.282743 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.282716 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:57:07.286067 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.286037 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf"] Jul 29 14:57:07.435173 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.435135 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e9de9c-aa82-49c1-b7d4-19a4c361ebe2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf\" (UID: \"20e9de9c-aa82-49c1-b7d4-19a4c361ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" Jul 29 14:57:07.536026 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.535912 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e9de9c-aa82-49c1-b7d4-19a4c361ebe2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf\" (UID: \"20e9de9c-aa82-49c1-b7d4-19a4c361ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" Jul 29 14:57:07.536599 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.536581 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e9de9c-aa82-49c1-b7d4-19a4c361ebe2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf\" (UID: \"20e9de9c-aa82-49c1-b7d4-19a4c361ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" Jul 29 14:57:07.595053 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.595019 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" Jul 29 14:57:07.751480 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:07.751455 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf"] Jul 29 14:57:07.753700 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:57:07.753669 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e9de9c_aa82_49c1_b7d4_19a4c361ebe2.slice/crio-bd06e19754fb5096dcaaac29c89bf38e634295980b8b14bdc573fb1c64c7e0d6 WatchSource:0}: Error finding container bd06e19754fb5096dcaaac29c89bf38e634295980b8b14bdc573fb1c64c7e0d6: Status 404 returned error can't find the container with id bd06e19754fb5096dcaaac29c89bf38e634295980b8b14bdc573fb1c64c7e0d6 Jul 29 14:57:08.176128 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:08.176082 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" event={"ID":"20e9de9c-aa82-49c1-b7d4-19a4c361ebe2","Type":"ContainerStarted","Data":"2287af2e9abc97d47b5b51349e01f9b12d65926e71f8ca201599622fe4a4824a"} Jul 29 14:57:08.176311 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:08.176149 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" event={"ID":"20e9de9c-aa82-49c1-b7d4-19a4c361ebe2","Type":"ContainerStarted","Data":"bd06e19754fb5096dcaaac29c89bf38e634295980b8b14bdc573fb1c64c7e0d6"} Jul 29 14:57:08.196043 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:08.195987 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" podStartSLOduration=1.195968744 podStartE2EDuration="1.195968744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:57:08.19231754 +0000 UTC m=+3361.393818712" watchObservedRunningTime="2026-07-29 14:57:08.195968744 +0000 UTC m=+3361.397469916" Jul 29 14:57:09.183875 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:09.183843 2560 generic.go:356] "Generic (PLEG): container finished" podID="20e9de9c-aa82-49c1-b7d4-19a4c361ebe2" containerID="2287af2e9abc97d47b5b51349e01f9b12d65926e71f8ca201599622fe4a4824a" exitCode=0 Jul 29 14:57:09.184298 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:09.183893 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" event={"ID":"20e9de9c-aa82-49c1-b7d4-19a4c361ebe2","Type":"ContainerDied","Data":"2287af2e9abc97d47b5b51349e01f9b12d65926e71f8ca201599622fe4a4824a"} Jul 29 14:57:10.338376 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:10.338343 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" Jul 29 14:57:10.463374 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:10.463274 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e9de9c-aa82-49c1-b7d4-19a4c361ebe2-maas-api-service-ca\") pod \"20e9de9c-aa82-49c1-b7d4-19a4c361ebe2\" (UID: \"20e9de9c-aa82-49c1-b7d4-19a4c361ebe2\") " Jul 29 14:57:10.463691 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:10.463668 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e9de9c-aa82-49c1-b7d4-19a4c361ebe2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "20e9de9c-aa82-49c1-b7d4-19a4c361ebe2" (UID: "20e9de9c-aa82-49c1-b7d4-19a4c361ebe2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:10.564928 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:10.564892 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20e9de9c-aa82-49c1-b7d4-19a4c361ebe2-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:11.198421 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:11.198386 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" Jul 29 14:57:11.198645 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:11.198426 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf" event={"ID":"20e9de9c-aa82-49c1-b7d4-19a4c361ebe2","Type":"ContainerDied","Data":"bd06e19754fb5096dcaaac29c89bf38e634295980b8b14bdc573fb1c64c7e0d6"} Jul 29 14:57:11.198645 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:11.198483 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd06e19754fb5096dcaaac29c89bf38e634295980b8b14bdc573fb1c64c7e0d6" Jul 29 14:57:17.894428 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:17.894372 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj"] Jul 29 14:57:17.897652 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:17.897623 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-61bc0a-29755605-kfpzj"] Jul 29 14:57:18.078666 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:18.078619 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf"] Jul 29 14:57:18.081646 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:18.081615 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-61bc0a-w8gmf"] Jul 29 14:57:19.381467 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:19.381428 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20e9de9c-aa82-49c1-b7d4-19a4c361ebe2" path="/var/lib/kubelet/pods/20e9de9c-aa82-49c1-b7d4-19a4c361ebe2/volumes" Jul 29 14:57:19.381910 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:19.381739 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2516203f-fde0-467e-824f-fbcdd8b20fdb" path="/var/lib/kubelet/pods/2516203f-fde0-467e-824f-fbcdd8b20fdb/volumes" Jul 29 14:57:21.444603 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:21.444567 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:57:24.981629 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:24.981584 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp"] Jul 29 14:57:24.982091 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:24.981858 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" containerID="cri-o://dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06" gracePeriod=30 Jul 29 14:57:25.408024 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.407930 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl"] Jul 29 14:57:25.408385 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.408371 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e9de9c-aa82-49c1-b7d4-19a4c361ebe2" containerName="revoke-keys" Jul 29 14:57:25.408431 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.408387 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e9de9c-aa82-49c1-b7d4-19a4c361ebe2" containerName="revoke-keys" Jul 29 14:57:25.408484 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.408475 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e9de9c-aa82-49c1-b7d4-19a4c361ebe2" containerName="revoke-keys" Jul 29 14:57:25.412868 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.412848 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" Jul 29 14:57:25.415515 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.415486 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8qrg\"" Jul 29 14:57:25.419660 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.419632 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl"] Jul 29 14:57:25.519014 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.518974 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dddc30-db15-4365-b4af-1cc678b56454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl\" (UID: \"92dddc30-db15-4365-b4af-1cc678b56454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" Jul 29 14:57:25.620469 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.620422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dddc30-db15-4365-b4af-1cc678b56454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl\" (UID: \"92dddc30-db15-4365-b4af-1cc678b56454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" Jul 29 14:57:25.621094 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.621075 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dddc30-db15-4365-b4af-1cc678b56454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl\" (UID: \"92dddc30-db15-4365-b4af-1cc678b56454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" Jul 29 14:57:25.725777 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.725734 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" Jul 29 14:57:25.877717 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:25.877690 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl"] Jul 29 14:57:25.878655 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:57:25.878621 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92dddc30_db15_4365_b4af_1cc678b56454.slice/crio-9a0cef9d1bed18a4701baebbaa1a5bd4e57d85fd56f7a22bd8cf468d3dfdc177 WatchSource:0}: Error finding container 9a0cef9d1bed18a4701baebbaa1a5bd4e57d85fd56f7a22bd8cf468d3dfdc177: Status 404 returned error can't find the container with id 9a0cef9d1bed18a4701baebbaa1a5bd4e57d85fd56f7a22bd8cf468d3dfdc177 Jul 29 14:57:26.301573 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:26.301499 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" event={"ID":"92dddc30-db15-4365-b4af-1cc678b56454","Type":"ContainerStarted","Data":"1cdfe0bab6c38cc8d88f6fc29c879d392c26804efe9a294bb5ec42c5cbe550df"} Jul 29 14:57:26.302039 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:26.301585 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" event={"ID":"92dddc30-db15-4365-b4af-1cc678b56454","Type":"ContainerStarted","Data":"9a0cef9d1bed18a4701baebbaa1a5bd4e57d85fd56f7a22bd8cf468d3dfdc177"} Jul 29 14:57:26.327044 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:26.326983 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" podStartSLOduration=1.326967124 podStartE2EDuration="1.326967124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:57:26.324240612 +0000 UTC m=+3379.525741779" watchObservedRunningTime="2026-07-29 14:57:26.326967124 +0000 UTC m=+3379.528468296" Jul 29 14:57:27.310813 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:27.310746 2560 generic.go:356] "Generic (PLEG): container finished" podID="92dddc30-db15-4365-b4af-1cc678b56454" containerID="1cdfe0bab6c38cc8d88f6fc29c879d392c26804efe9a294bb5ec42c5cbe550df" exitCode=0 Jul 29 14:57:27.310813 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:27.310813 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" event={"ID":"92dddc30-db15-4365-b4af-1cc678b56454","Type":"ContainerDied","Data":"1cdfe0bab6c38cc8d88f6fc29c879d392c26804efe9a294bb5ec42c5cbe550df"} Jul 29 14:57:28.472494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:28.472459 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" Jul 29 14:57:28.550940 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:28.550897 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dddc30-db15-4365-b4af-1cc678b56454-maas-api-service-ca\") pod \"92dddc30-db15-4365-b4af-1cc678b56454\" (UID: \"92dddc30-db15-4365-b4af-1cc678b56454\") " Jul 29 14:57:28.551334 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:28.551308 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dddc30-db15-4365-b4af-1cc678b56454-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92dddc30-db15-4365-b4af-1cc678b56454" (UID: "92dddc30-db15-4365-b4af-1cc678b56454"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:28.652921 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:28.652818 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dddc30-db15-4365-b4af-1cc678b56454-maas-api-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:29.333797 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:29.333761 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" Jul 29 14:57:29.333981 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:29.333794 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl" event={"ID":"92dddc30-db15-4365-b4af-1cc678b56454","Type":"ContainerDied","Data":"9a0cef9d1bed18a4701baebbaa1a5bd4e57d85fd56f7a22bd8cf468d3dfdc177"} Jul 29 14:57:29.333981 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:29.333869 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a0cef9d1bed18a4701baebbaa1a5bd4e57d85fd56f7a22bd8cf468d3dfdc177" Jul 29 14:57:30.254065 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.253995 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:57:30.341937 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.341907 2560 generic.go:356] "Generic (PLEG): container finished" podID="629f2947-c498-4d52-923b-179e9bd599c8" containerID="dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06" exitCode=0 Jul 29 14:57:30.342133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.341946 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" event={"ID":"629f2947-c498-4d52-923b-179e9bd599c8","Type":"ContainerDied","Data":"dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06"} Jul 29 14:57:30.342133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.341966 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" Jul 29 14:57:30.342133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.341994 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" event={"ID":"629f2947-c498-4d52-923b-179e9bd599c8","Type":"ContainerDied","Data":"096ad0be2493818e986028807bd2a288bf51ff979b52bbe6427206b4c5c86777"} Jul 29 14:57:30.342133 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.342016 2560 scope.go:117] "RemoveContainer" containerID="dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06" Jul 29 14:57:30.351907 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.351887 2560 scope.go:117] "RemoveContainer" containerID="42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae" Jul 29 14:57:30.363193 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.363174 2560 scope.go:117] "RemoveContainer" containerID="dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06" Jul 29 14:57:30.363500 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:57:30.363478 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06\": container with ID starting with dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06 not found: ID does not exist" containerID="dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06" Jul 29 14:57:30.363576 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.363512 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06"} err="failed to get container status \"dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06\": rpc error: code = NotFound desc = could not find container \"dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06\": container with ID starting with dadd2609cd4b0df9df1293eec6ac953c5788ccf15c68b283da4de8da7b5d8b06 not found: ID does not exist" Jul 29 14:57:30.363576 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.363532 2560 scope.go:117] "RemoveContainer" containerID="42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae" Jul 29 14:57:30.363825 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:57:30.363805 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae\": container with ID starting with 42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae not found: ID does not exist" containerID="42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae" Jul 29 14:57:30.363869 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.363834 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae"} err="failed to get container status \"42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae\": rpc error: code = NotFound desc = could not find container \"42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae\": container with ID starting with 42732445f9df3ed5e79902489be22d7fed62110fe9847ff5285db5ebb9bb1aae not found: ID does not exist" Jul 29 14:57:30.369169 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369148 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629f2947-c498-4d52-923b-179e9bd599c8-istiod-ca-cert\") pod \"629f2947-c498-4d52-923b-179e9bd599c8\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " Jul 29 14:57:30.369262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369178 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629f2947-c498-4d52-923b-179e9bd599c8-istio-podinfo\") pod \"629f2947-c498-4d52-923b-179e9bd599c8\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " Jul 29 14:57:30.369262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369206 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-certs\") pod \"629f2947-c498-4d52-923b-179e9bd599c8\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " Jul 29 14:57:30.369262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369243 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n55tg\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-kube-api-access-n55tg\") pod \"629f2947-c498-4d52-923b-179e9bd599c8\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " Jul 29 14:57:30.369370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369275 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-data\") pod \"629f2947-c498-4d52-923b-179e9bd599c8\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " Jul 29 14:57:30.369370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369307 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-envoy\") pod \"629f2947-c498-4d52-923b-179e9bd599c8\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " Jul 29 14:57:30.369370 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369348 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-socket\") pod \"629f2947-c498-4d52-923b-179e9bd599c8\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " Jul 29 14:57:30.369518 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369391 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-istio-token\") pod \"629f2947-c498-4d52-923b-179e9bd599c8\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " Jul 29 14:57:30.369518 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369411 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-credential-socket\") pod \"629f2947-c498-4d52-923b-179e9bd599c8\" (UID: \"629f2947-c498-4d52-923b-179e9bd599c8\") " Jul 29 14:57:30.369663 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369589 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629f2947-c498-4d52-923b-179e9bd599c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "629f2947-c498-4d52-923b-179e9bd599c8" (UID: "629f2947-c498-4d52-923b-179e9bd599c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:30.369663 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369644 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "629f2947-c498-4d52-923b-179e9bd599c8" (UID: "629f2947-c498-4d52-923b-179e9bd599c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:57:30.369774 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369693 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "629f2947-c498-4d52-923b-179e9bd599c8" (UID: "629f2947-c498-4d52-923b-179e9bd599c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:57:30.370018 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369944 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629f2947-c498-4d52-923b-179e9bd599c8-istiod-ca-cert\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:30.370018 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369971 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-certs\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:30.370018 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.369984 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-workload-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:30.370414 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.370061 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "629f2947-c498-4d52-923b-179e9bd599c8" (UID: "629f2947-c498-4d52-923b-179e9bd599c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:57:30.370673 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.370648 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "629f2947-c498-4d52-923b-179e9bd599c8" (UID: "629f2947-c498-4d52-923b-179e9bd599c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:57:30.371832 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.371806 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "629f2947-c498-4d52-923b-179e9bd599c8" (UID: "629f2947-c498-4d52-923b-179e9bd599c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:30.371955 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.371894 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-kube-api-access-n55tg" (OuterVolumeSpecName: "kube-api-access-n55tg") pod "629f2947-c498-4d52-923b-179e9bd599c8" (UID: "629f2947-c498-4d52-923b-179e9bd599c8"). InnerVolumeSpecName "kube-api-access-n55tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:30.372109 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.372092 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/629f2947-c498-4d52-923b-179e9bd599c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "629f2947-c498-4d52-923b-179e9bd599c8" (UID: "629f2947-c498-4d52-923b-179e9bd599c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 14:57:30.372262 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.372241 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "629f2947-c498-4d52-923b-179e9bd599c8" (UID: "629f2947-c498-4d52-923b-179e9bd599c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 14:57:30.470719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.470680 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-istio-token\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:30.470719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.470713 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-credential-socket\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:30.470719 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.470724 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629f2947-c498-4d52-923b-179e9bd599c8-istio-podinfo\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:30.470982 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.470733 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n55tg\" (UniqueName: \"kubernetes.io/projected/629f2947-c498-4d52-923b-179e9bd599c8-kube-api-access-n55tg\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:30.470982 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.470742 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-data\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:30.470982 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.470750 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629f2947-c498-4d52-923b-179e9bd599c8-istio-envoy\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:30.668281 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.668248 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp"] Jul 29 14:57:30.671134 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:30.671102 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp"] Jul 29 14:57:31.218854 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:31.218808 2560 patch_prober.go:28] interesting pod/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 14:57:31.219131 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:31.218895 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-61bc0a-openshift-default-66594c764d-8whfp" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 14:57:31.381832 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:31.381798 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="629f2947-c498-4d52-923b-179e9bd599c8" path="/var/lib/kubelet/pods/629f2947-c498-4d52-923b-179e9bd599c8/volumes" Jul 29 14:57:35.860344 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:35.860308 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5"] Jul 29 14:57:35.860900 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:35.860559 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" podUID="cae99d8b-2fee-494b-bd27-d60af6b8f4e5" containerName="maas-api" containerID="cri-o://9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298" gracePeriod=30 Jul 29 14:57:35.896250 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:35.896179 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk"] Jul 29 14:57:35.899077 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:35.899047 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-532c20-29755605-zksrk"] Jul 29 14:57:36.110011 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.109971 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl"] Jul 29 14:57:36.114222 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.114143 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-532c20-zhlgl"] Jul 29 14:57:36.141496 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.141470 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:57:36.230277 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.230242 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-openshift-service-ca\") pod \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " Jul 29 14:57:36.230484 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.230290 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-maas-api-tls\") pod \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " Jul 29 14:57:36.230484 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.230322 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kswp\" (UniqueName: \"kubernetes.io/projected/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-kube-api-access-9kswp\") pod \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\" (UID: \"cae99d8b-2fee-494b-bd27-d60af6b8f4e5\") " Jul 29 14:57:36.230733 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.230699 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cae99d8b-2fee-494b-bd27-d60af6b8f4e5" (UID: "cae99d8b-2fee-494b-bd27-d60af6b8f4e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 14:57:36.232753 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.232726 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cae99d8b-2fee-494b-bd27-d60af6b8f4e5" (UID: "cae99d8b-2fee-494b-bd27-d60af6b8f4e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 14:57:36.232884 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.232752 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-kube-api-access-9kswp" (OuterVolumeSpecName: "kube-api-access-9kswp") pod "cae99d8b-2fee-494b-bd27-d60af6b8f4e5" (UID: "cae99d8b-2fee-494b-bd27-d60af6b8f4e5"). InnerVolumeSpecName "kube-api-access-9kswp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 14:57:36.332141 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.332103 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-openshift-service-ca\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:36.332141 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.332141 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-maas-api-tls\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:36.332350 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.332156 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kswp\" (UniqueName: \"kubernetes.io/projected/cae99d8b-2fee-494b-bd27-d60af6b8f4e5-kube-api-access-9kswp\") on node \"ip-10-0-139-155.ec2.internal\" DevicePath \"\"" Jul 29 14:57:36.387001 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.386910 2560 generic.go:356] "Generic (PLEG): container finished" podID="cae99d8b-2fee-494b-bd27-d60af6b8f4e5" containerID="9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298" exitCode=0 Jul 29 14:57:36.387001 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.386958 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" Jul 29 14:57:36.387001 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.386989 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" event={"ID":"cae99d8b-2fee-494b-bd27-d60af6b8f4e5","Type":"ContainerDied","Data":"9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298"} Jul 29 14:57:36.387232 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.387051 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5" event={"ID":"cae99d8b-2fee-494b-bd27-d60af6b8f4e5","Type":"ContainerDied","Data":"bbd90d15fb6f92cf4613716fddbbf0808d8f3a103e8625c8e4ff93165acaa2f8"} Jul 29 14:57:36.387232 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.387075 2560 scope.go:117] "RemoveContainer" containerID="9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298" Jul 29 14:57:36.398561 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.398508 2560 scope.go:117] "RemoveContainer" containerID="9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298" Jul 29 14:57:36.398898 ip-10-0-139-155 kubenswrapper[2560]: E0729 14:57:36.398876 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298\": container with ID starting with 9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298 not found: ID does not exist" containerID="9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298" Jul 29 14:57:36.398963 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.398910 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298"} err="failed to get container status \"9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298\": rpc error: code = NotFound desc = could not find container \"9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298\": container with ID starting with 9c4510e8038844dd3bb594beb84b202f1b3178f41e9f8dc2e894f8bcfe890298 not found: ID does not exist" Jul 29 14:57:36.411912 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.411880 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5"] Jul 29 14:57:36.414606 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:36.414580 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-532c20-646cbc49b7-gc4q5"] Jul 29 14:57:37.381773 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:37.381739 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92dddc30-db15-4365-b4af-1cc678b56454" path="/var/lib/kubelet/pods/92dddc30-db15-4365-b4af-1cc678b56454/volumes" Jul 29 14:57:37.382164 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:37.382037 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c651dc16-5f49-4ee3-888d-553eccfd48d4" path="/var/lib/kubelet/pods/c651dc16-5f49-4ee3-888d-553eccfd48d4/volumes" Jul 29 14:57:37.382337 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:37.382325 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae99d8b-2fee-494b-bd27-d60af6b8f4e5" path="/var/lib/kubelet/pods/cae99d8b-2fee-494b-bd27-d60af6b8f4e5/volumes" Jul 29 14:57:39.356662 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:39.356627 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jrpx2"] Jul 29 14:57:43.417600 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:43.417569 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-758465fbff-nb9c6_29c73cb9-ff52-4c18-9aca-948a80c74419/authorino/0.log" Jul 29 14:57:49.003158 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:49.003117 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ldbdm_ba573e67-60f3-4305-8459-e91d880b01a7/manager/0.log" Jul 29 14:57:49.134475 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:49.134442 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zrfzs_dff5f71f-1475-46e0-a2eb-e7676472a8ea/manager/0.log" Jul 29 14:57:49.529199 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:49.529165 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mhpqh_77b01920-4ef0-42f1-b160-c11a12fc72c8/manager/0.log" Jul 29 14:57:49.659378 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:49.659343 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9_79a4777a-ea6f-4969-ab7e-5a18b5c253cd/manager/0.log" Jul 29 14:57:49.676085 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:49.676050 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9_79a4777a-ea6f-4969-ab7e-5a18b5c253cd/copy-manifests/0.log" Jul 29 14:57:50.425966 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:50.425928 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755575-c52sl_bbdbc566-063c-4909-b252-849d94e32bb1/cleanup/0.log" Jul 29 14:57:50.536921 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:50.536894 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755590-9mlqn_db203d13-bcb5-4386-8667-09166e8b6b48/cleanup/0.log" Jul 29 14:57:50.648909 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:50.648879 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755605-frxbx_2032a639-972d-44eb-9339-807e9b5f8bc9/cleanup/0.log" Jul 29 14:57:50.758224 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:50.758128 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-nk7rq_3df5a58a-b2dc-4e91-9b7b-d082b01d2acc/postgres/0.log" Jul 29 14:57:51.602298 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:51.602265 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556_3301ee40-54e0-47b9-94f1-357d7763fb4f/util/0.log" Jul 29 14:57:51.611443 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:51.611417 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556_3301ee40-54e0-47b9-94f1-357d7763fb4f/pull/0.log" Jul 29 14:57:51.618139 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:51.618101 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556_3301ee40-54e0-47b9-94f1-357d7763fb4f/extract/0.log" Jul 29 14:57:51.734839 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:51.734803 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh_fb0151f9-7a05-44e3-b38a-3d6d96b7363e/util/0.log" Jul 29 14:57:51.742774 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:51.742743 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh_fb0151f9-7a05-44e3-b38a-3d6d96b7363e/pull/0.log" Jul 29 14:57:51.750280 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:51.750258 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh_fb0151f9-7a05-44e3-b38a-3d6d96b7363e/extract/0.log" Jul 29 14:57:51.875042 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:51.874944 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm_96d924bb-eacb-4e53-a8f8-89fe595a9d69/util/0.log" Jul 29 14:57:51.882904 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:51.882872 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm_96d924bb-eacb-4e53-a8f8-89fe595a9d69/pull/0.log" Jul 29 14:57:51.891005 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:51.890977 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm_96d924bb-eacb-4e53-a8f8-89fe595a9d69/extract/0.log" Jul 29 14:57:52.007168 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.007135 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx_839a6e05-d66f-4844-a6b6-144db923302b/util/0.log" Jul 29 14:57:52.016053 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.016028 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx_839a6e05-d66f-4844-a6b6-144db923302b/pull/0.log" Jul 29 14:57:52.023822 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.023787 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx_839a6e05-d66f-4844-a6b6-144db923302b/extract/0.log" Jul 29 14:57:52.155033 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.154943 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-758465fbff-nb9c6_29c73cb9-ff52-4c18-9aca-948a80c74419/authorino/0.log" Jul 29 14:57:52.281337 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.281305 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-rkqhb_27b7073f-03b6-41aa-8f19-6d5b71ef82bf/manager/0.log" Jul 29 14:57:52.391135 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.391101 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gbqkt_c5017b6c-3682-44ab-9e76-8d3daea4c58b/manager/0.log" Jul 29 14:57:52.501480 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.501453 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bg5c6_cd43c72b-7401-4f5e-b3a3-de3dae69014d/kuadrant-console-plugin/0.log" Jul 29 14:57:52.627416 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.627386 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rn5jh_b810810c-028e-473e-b761-7e1db11b727a/registry-server/0.log" Jul 29 14:57:52.758989 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.758878 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hxrkl_8f255ca0-8fcf-4b0d-a789-77bd878631f0/manager/0.log" Jul 29 14:57:52.872623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:52.872589 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jrpx2_21ae4806-319b-4ddc-a6ce-f5756085e0ac/limitador/0.log" Jul 29 14:57:53.681375 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:53.681339 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b4bcbd496-v65rv_20eadaef-47b1-46fb-b47d-d9ca8d1f8931/kube-auth-proxy/0.log" Jul 29 14:57:53.812360 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:53.812328 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zv49l_cab85095-f633-4d39-aebd-3716781e6c66/istio-proxy/1.log" Jul 29 14:57:54.166185 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:54.166093 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78f77ccbf5-cjvmf_e04e6603-8385-4ffb-aa85-ccae91e7658f/router/0.log" Jul 29 14:57:54.626136 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:57:54.626107 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-l7fqw_a95663aa-9693-4fd4-9621-1222214c08d1/main/0.log" Jul 29 14:58:00.633387 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.633294 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tpxzz/must-gather-q72k5"] Jul 29 14:58:00.633950 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.633912 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae99d8b-2fee-494b-bd27-d60af6b8f4e5" containerName="maas-api" Jul 29 14:58:00.633950 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.633931 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae99d8b-2fee-494b-bd27-d60af6b8f4e5" containerName="maas-api" Jul 29 14:58:00.633950 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.633948 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92dddc30-db15-4365-b4af-1cc678b56454" containerName="revoke-keys" Jul 29 14:58:00.634073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.633956 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="92dddc30-db15-4365-b4af-1cc678b56454" containerName="revoke-keys" Jul 29 14:58:00.634073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.633971 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" Jul 29 14:58:00.634073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.633980 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" Jul 29 14:58:00.634073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.633988 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" Jul 29 14:58:00.634073 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.633993 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" Jul 29 14:58:00.634224 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.634078 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae99d8b-2fee-494b-bd27-d60af6b8f4e5" containerName="maas-api" Jul 29 14:58:00.634224 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.634088 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="92dddc30-db15-4365-b4af-1cc678b56454" containerName="revoke-keys" Jul 29 14:58:00.634224 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.634094 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" Jul 29 14:58:00.634224 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.634105 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="629f2947-c498-4d52-923b-179e9bd599c8" containerName="istio-proxy" Jul 29 14:58:00.637567 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.637525 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpxzz/must-gather-q72k5" Jul 29 14:58:00.640015 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.639993 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpxzz\"/\"openshift-service-ca.crt\"" Jul 29 14:58:00.640768 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.640749 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpxzz\"/\"kube-root-ca.crt\"" Jul 29 14:58:00.640881 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.640782 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tpxzz\"/\"default-dockercfg-2v4xz\"" Jul 29 14:58:00.645968 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.645762 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpxzz/must-gather-q72k5"] Jul 29 14:58:00.658978 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.658945 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkm2\" (UniqueName: \"kubernetes.io/projected/f117feaf-28ad-4c78-8312-69e5077676bf-kube-api-access-zbkm2\") pod \"must-gather-q72k5\" (UID: \"f117feaf-28ad-4c78-8312-69e5077676bf\") " pod="openshift-must-gather-tpxzz/must-gather-q72k5" Jul 29 14:58:00.659173 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.659077 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f117feaf-28ad-4c78-8312-69e5077676bf-must-gather-output\") pod \"must-gather-q72k5\" (UID: \"f117feaf-28ad-4c78-8312-69e5077676bf\") " pod="openshift-must-gather-tpxzz/must-gather-q72k5" Jul 29 14:58:00.759752 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.759715 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f117feaf-28ad-4c78-8312-69e5077676bf-must-gather-output\") pod \"must-gather-q72k5\" (UID: \"f117feaf-28ad-4c78-8312-69e5077676bf\") " pod="openshift-must-gather-tpxzz/must-gather-q72k5" Jul 29 14:58:00.759946 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.759773 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkm2\" (UniqueName: \"kubernetes.io/projected/f117feaf-28ad-4c78-8312-69e5077676bf-kube-api-access-zbkm2\") pod \"must-gather-q72k5\" (UID: \"f117feaf-28ad-4c78-8312-69e5077676bf\") " pod="openshift-must-gather-tpxzz/must-gather-q72k5" Jul 29 14:58:00.760060 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.760042 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f117feaf-28ad-4c78-8312-69e5077676bf-must-gather-output\") pod \"must-gather-q72k5\" (UID: \"f117feaf-28ad-4c78-8312-69e5077676bf\") " pod="openshift-must-gather-tpxzz/must-gather-q72k5" Jul 29 14:58:00.769324 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.769301 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkm2\" (UniqueName: \"kubernetes.io/projected/f117feaf-28ad-4c78-8312-69e5077676bf-kube-api-access-zbkm2\") pod \"must-gather-q72k5\" (UID: \"f117feaf-28ad-4c78-8312-69e5077676bf\") " pod="openshift-must-gather-tpxzz/must-gather-q72k5" Jul 29 14:58:00.948932 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:00.948896 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpxzz/must-gather-q72k5" Jul 29 14:58:01.101691 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:01.101661 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpxzz/must-gather-q72k5"] Jul 29 14:58:01.102832 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:58:01.102806 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf117feaf_28ad_4c78_8312_69e5077676bf.slice/crio-f36a1a46b9b5ac7d699b2fe48a0489d670aa778f5bb48c40cce3842c8a14585f WatchSource:0}: Error finding container f36a1a46b9b5ac7d699b2fe48a0489d670aa778f5bb48c40cce3842c8a14585f: Status 404 returned error can't find the container with id f36a1a46b9b5ac7d699b2fe48a0489d670aa778f5bb48c40cce3842c8a14585f Jul 29 14:58:01.104841 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:01.104823 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 14:58:01.560000 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:01.559966 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpxzz/must-gather-q72k5" event={"ID":"f117feaf-28ad-4c78-8312-69e5077676bf","Type":"ContainerStarted","Data":"f36a1a46b9b5ac7d699b2fe48a0489d670aa778f5bb48c40cce3842c8a14585f"} Jul 29 14:58:02.572459 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:02.572406 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpxzz/must-gather-q72k5" event={"ID":"f117feaf-28ad-4c78-8312-69e5077676bf","Type":"ContainerStarted","Data":"1826199e63ded4813c42e23de15800882eec5f8d5a9ac975b5dd2639d852e7a3"} Jul 29 14:58:02.572954 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:02.572467 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpxzz/must-gather-q72k5" event={"ID":"f117feaf-28ad-4c78-8312-69e5077676bf","Type":"ContainerStarted","Data":"1a66a3181c5cb40d3840232c1b212ad3a71b2535b021037f88eae551d650b92b"} Jul 29 14:58:02.590864 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:02.590809 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tpxzz/must-gather-q72k5" podStartSLOduration=1.771707927 podStartE2EDuration="2.590792095s" totalImagesPullingTime="819.084168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:58:02.58931631 +0000 UTC m=+3415.790817483" watchObservedRunningTime="2026-07-29 14:58:02.590792095 +0000 UTC m=+3415.792293266" Jul 29 14:58:03.536130 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:03.536089 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2vm4f_6ebe845f-c3d6-4eed-a05e-88b608cc827f/global-pull-secret-syncer/0.log" Jul 29 14:58:03.755285 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:03.755249 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hpg7p_d8c01759-3d25-4362-922e-dc595a65771e/konnectivity-agent/0.log" Jul 29 14:58:03.832867 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:03.832783 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-155.ec2.internal_5cca166521caabdf057abdac110b1500/haproxy/0.log" Jul 29 14:58:08.101626 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:08.101583 2560 scope.go:117] "RemoveContainer" containerID="19a62744b129cc14890a96d901a877280a6dbb6e91bb3e38ae3c44676e3be293" Jul 29 14:58:08.117965 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:08.117927 2560 scope.go:117] "RemoveContainer" containerID="903780d172de659a6a0b3d80106867db70a8740eaa0416fae7e0d29c264981f8" Jul 29 14:58:08.783251 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:08.783216 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556_3301ee40-54e0-47b9-94f1-357d7763fb4f/extract/0.log" Jul 29 14:58:08.811959 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:08.811925 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556_3301ee40-54e0-47b9-94f1-357d7763fb4f/util/0.log" Jul 29 14:58:08.856171 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:08.856130 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kz556_3301ee40-54e0-47b9-94f1-357d7763fb4f/pull/0.log" Jul 29 14:58:08.890364 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:08.890329 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh_fb0151f9-7a05-44e3-b38a-3d6d96b7363e/extract/0.log" Jul 29 14:58:08.916526 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:08.916491 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh_fb0151f9-7a05-44e3-b38a-3d6d96b7363e/util/0.log" Jul 29 14:58:08.952383 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:08.952349 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0txbzh_fb0151f9-7a05-44e3-b38a-3d6d96b7363e/pull/0.log" Jul 29 14:58:08.991092 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:08.991056 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm_96d924bb-eacb-4e53-a8f8-89fe595a9d69/extract/0.log" Jul 29 14:58:09.019644 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.019610 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm_96d924bb-eacb-4e53-a8f8-89fe595a9d69/util/0.log" Jul 29 14:58:09.046911 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.046820 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ndmbm_96d924bb-eacb-4e53-a8f8-89fe595a9d69/pull/0.log" Jul 29 14:58:09.079368 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.079327 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx_839a6e05-d66f-4844-a6b6-144db923302b/extract/0.log" Jul 29 14:58:09.106303 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.106270 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx_839a6e05-d66f-4844-a6b6-144db923302b/util/0.log" Jul 29 14:58:09.137263 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.137175 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtmrx_839a6e05-d66f-4844-a6b6-144db923302b/pull/0.log" Jul 29 14:58:09.746584 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.746387 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-758465fbff-nb9c6_29c73cb9-ff52-4c18-9aca-948a80c74419/authorino/0.log" Jul 29 14:58:09.788569 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.788376 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-rkqhb_27b7073f-03b6-41aa-8f19-6d5b71ef82bf/manager/0.log" Jul 29 14:58:09.826447 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.826404 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gbqkt_c5017b6c-3682-44ab-9e76-8d3daea4c58b/manager/0.log" Jul 29 14:58:09.860444 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.860409 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bg5c6_cd43c72b-7401-4f5e-b3a3-de3dae69014d/kuadrant-console-plugin/0.log" Jul 29 14:58:09.928805 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:09.928763 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rn5jh_b810810c-028e-473e-b761-7e1db11b727a/registry-server/0.log" Jul 29 14:58:10.303593 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:10.303523 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hxrkl_8f255ca0-8fcf-4b0d-a789-77bd878631f0/manager/0.log" Jul 29 14:58:10.332288 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:10.332264 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jrpx2_21ae4806-319b-4ddc-a6ce-f5756085e0ac/limitador/0.log" Jul 29 14:58:12.161494 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.161456 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c/alertmanager/0.log" Jul 29 14:58:12.189137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.189107 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c/config-reloader/0.log" Jul 29 14:58:12.215209 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.215173 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c/kube-rbac-proxy-web/0.log" Jul 29 14:58:12.241374 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.241291 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c/kube-rbac-proxy/0.log" Jul 29 14:58:12.268950 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.268921 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c/kube-rbac-proxy-metric/0.log" Jul 29 14:58:12.293142 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.293098 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c/prom-label-proxy/0.log" Jul 29 14:58:12.320283 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.320249 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7976e3de-69fd-4a4c-8c26-fc92fcbcaa9c/init-config-reloader/0.log" Jul 29 14:58:12.375342 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.375298 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-xfxm6_a48e2b71-ac38-43d0-8842-9c4e2c9cd833/cluster-monitoring-operator/0.log" Jul 29 14:58:12.509524 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.509498 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-2ghhb_9e896d9b-f6a9-4f21-b9f0-e05deb1315c0/monitoring-plugin/0.log" Jul 29 14:58:12.629679 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.629645 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lbzn6_67b41abc-c4f0-4424-8eec-aaebbfec736a/node-exporter/0.log" Jul 29 14:58:12.664024 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.663997 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lbzn6_67b41abc-c4f0-4424-8eec-aaebbfec736a/kube-rbac-proxy/0.log" Jul 29 14:58:12.686772 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:12.686736 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lbzn6_67b41abc-c4f0-4424-8eec-aaebbfec736a/init-textfile/0.log" Jul 29 14:58:13.069305 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.069269 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6xvf2_48b47b5f-c165-4452-b079-5c96b6a7fb3f/prometheus-operator/0.log" Jul 29 14:58:13.088348 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.088315 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6xvf2_48b47b5f-c165-4452-b079-5c96b6a7fb3f/kube-rbac-proxy/0.log" Jul 29 14:58:13.239452 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.239418 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/thanos-query/0.log" Jul 29 14:58:13.265502 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.265470 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/kube-rbac-proxy-web/0.log" Jul 29 14:58:13.291310 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.291279 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/kube-rbac-proxy/0.log" Jul 29 14:58:13.313878 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.313844 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/prom-label-proxy/0.log" Jul 29 14:58:13.336332 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.336245 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/kube-rbac-proxy-rules/0.log" Jul 29 14:58:13.360988 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.360950 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-558b99f77-772nh_bef742fe-a63b-4f80-bfd5-0435ce451118/kube-rbac-proxy-metrics/0.log" Jul 29 14:58:13.988137 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.988099 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m"] Jul 29 14:58:13.996563 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:13.996515 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.010432 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.010403 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m"] Jul 29 14:58:14.099623 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.099590 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-proc\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.099820 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.099634 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-sys\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.099820 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.099771 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-lib-modules\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.099967 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.099847 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-podres\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.099967 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.099928 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sncw6\" (UniqueName: \"kubernetes.io/projected/0013dc84-9e58-47fe-9503-6163659b0bf2-kube-api-access-sncw6\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.200810 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.200766 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-proc\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.201007 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.200821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-sys\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.201007 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.200883 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-lib-modules\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.201007 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.200931 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-podres\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.201007 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.201000 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sncw6\" (UniqueName: \"kubernetes.io/projected/0013dc84-9e58-47fe-9503-6163659b0bf2-kube-api-access-sncw6\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.201479 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.201437 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-proc\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.201588 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.201510 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-sys\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.201663 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.201648 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-lib-modules\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.201738 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.201720 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0013dc84-9e58-47fe-9503-6163659b0bf2-podres\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.210094 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.210055 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sncw6\" (UniqueName: \"kubernetes.io/projected/0013dc84-9e58-47fe-9503-6163659b0bf2-kube-api-access-sncw6\") pod \"perf-node-gather-daemonset-5cl2m\" (UID: \"0013dc84-9e58-47fe-9503-6163659b0bf2\") " pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.314247 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.314160 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.524043 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.522868 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m"] Jul 29 14:58:14.525161 ip-10-0-139-155 kubenswrapper[2560]: W0729 14:58:14.525126 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0013dc84_9e58_47fe_9503_6163659b0bf2.slice/crio-0c44581fcb3c1edc929c9b607781e0c054665b835216e2d589ec1811bf2c52c9 WatchSource:0}: Error finding container 0c44581fcb3c1edc929c9b607781e0c054665b835216e2d589ec1811bf2c52c9: Status 404 returned error can't find the container with id 0c44581fcb3c1edc929c9b607781e0c054665b835216e2d589ec1811bf2c52c9 Jul 29 14:58:14.706987 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.706946 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" event={"ID":"0013dc84-9e58-47fe-9503-6163659b0bf2","Type":"ContainerStarted","Data":"f57d3e0c5bdd7cf5a7061b2beeb473a625b43c2ad0dab3c41d373e958b76f29e"} Jul 29 14:58:14.707172 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.707019 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" event={"ID":"0013dc84-9e58-47fe-9503-6163659b0bf2","Type":"ContainerStarted","Data":"0c44581fcb3c1edc929c9b607781e0c054665b835216e2d589ec1811bf2c52c9"} Jul 29 14:58:14.707241 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.707187 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:14.726892 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:14.726840 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" podStartSLOduration=1.726821099 podStartE2EDuration="1.726821099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:58:14.72483358 +0000 UTC m=+3427.926334753" watchObservedRunningTime="2026-07-29 14:58:14.726821099 +0000 UTC m=+3427.928322271" Jul 29 14:58:15.124607 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:15.124571 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/2.log" Jul 29 14:58:15.131341 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:15.131305 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zn2dp_b63ed94d-a128-4c71-8465-479c86c3cc34/console-operator/3.log" Jul 29 14:58:15.636437 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:15.636410 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d45d947f5-hz9nk_a26c1bb4-437c-43e0-b9b4-98be46a85954/console/0.log" Jul 29 14:58:16.222941 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:16.222916 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-p5n9n_ca7f7b51-a0e9-444f-8360-02e1976a84d6/volume-data-source-validator/0.log" Jul 29 14:58:17.427497 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:17.425629 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sbgpj_7010f78e-9440-452f-9616-8abac941737b/dns/0.log" Jul 29 14:58:17.452619 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:17.452588 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sbgpj_7010f78e-9440-452f-9616-8abac941737b/kube-rbac-proxy/0.log" Jul 29 14:58:17.500999 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:17.500973 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w5jp9_19853864-4d88-4abe-8485-aa693e06ab08/dns-node-resolver/0.log" Jul 29 14:58:18.131435 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:18.131392 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v5hwn_e9a1dbd3-0eba-4141-a243-bfb4b7da0740/node-ca/0.log" Jul 29 14:58:19.349396 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:19.349364 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b4bcbd496-v65rv_20eadaef-47b1-46fb-b47d-d9ca8d1f8931/kube-auth-proxy/0.log" Jul 29 14:58:19.382309 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:19.382280 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zv49l_cab85095-f633-4d39-aebd-3716781e6c66/istio-proxy/0.log" Jul 29 14:58:19.441381 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:19.441347 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zv49l_cab85095-f633-4d39-aebd-3716781e6c66/istio-proxy/1.log" Jul 29 14:58:19.537236 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:19.537204 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78f77ccbf5-cjvmf_e04e6603-8385-4ffb-aa85-ccae91e7658f/router/0.log" Jul 29 14:58:20.410035 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:20.410000 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nm2x6_0c6e3418-a206-431f-aad9-c917dd9df3f4/serve-healthcheck-canary/0.log" Jul 29 14:58:20.729343 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:20.729314 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tpxzz/perf-node-gather-daemonset-5cl2m" Jul 29 14:58:20.889385 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:20.889340 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-pt4nq_58f1ae35-67f6-46a3-9a41-7cd808fe739c/insights-operator/0.log" Jul 29 14:58:20.894116 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:20.894083 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-pt4nq_58f1ae35-67f6-46a3-9a41-7cd808fe739c/insights-operator/1.log" Jul 29 14:58:21.003467 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:21.003385 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5b4fk_1d771142-7766-4ff7-8a28-4b3eea8bdfbd/kube-rbac-proxy/0.log" Jul 29 14:58:21.029760 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:21.029732 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5b4fk_1d771142-7766-4ff7-8a28-4b3eea8bdfbd/exporter/0.log" Jul 29 14:58:21.055359 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:21.055326 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5b4fk_1d771142-7766-4ff7-8a28-4b3eea8bdfbd/extractor/0.log" Jul 29 14:58:23.518572 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:23.518486 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ldbdm_ba573e67-60f3-4305-8459-e91d880b01a7/manager/0.log" Jul 29 14:58:23.971074 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:23.970918 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zrfzs_dff5f71f-1475-46e0-a2eb-e7676472a8ea/manager/0.log" Jul 29 14:58:24.252113 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:24.252081 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mhpqh_77b01920-4ef0-42f1-b160-c11a12fc72c8/manager/0.log" Jul 29 14:58:24.400249 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:24.400153 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9_79a4777a-ea6f-4969-ab7e-5a18b5c253cd/manager/0.log" Jul 29 14:58:24.417864 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:24.417835 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff7cbc5f5-l75s9_79a4777a-ea6f-4969-ab7e-5a18b5c253cd/copy-manifests/0.log" Jul 29 14:58:25.297966 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:25.297930 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b4bcbd496-v65rv_20eadaef-47b1-46fb-b47d-d9ca8d1f8931/kube-auth-proxy/0.log" Jul 29 14:58:25.323035 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:25.323001 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zv49l_cab85095-f633-4d39-aebd-3716781e6c66/istio-proxy/0.log" Jul 29 14:58:25.374069 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:25.374030 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zv49l_cab85095-f633-4d39-aebd-3716781e6c66/istio-proxy/1.log" Jul 29 14:58:25.452248 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:25.452215 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78f77ccbf5-cjvmf_e04e6603-8385-4ffb-aa85-ccae91e7658f/router/0.log" Jul 29 14:58:25.957235 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:25.957205 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lwjpl_275e0fbc-1694-4401-b84f-bf64cbe6eb78/openshift-lws-operator/0.log" Jul 29 14:58:26.999147 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:26.999105 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b4bcbd496-v65rv_20eadaef-47b1-46fb-b47d-d9ca8d1f8931/kube-auth-proxy/0.log" Jul 29 14:58:27.024893 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:27.024847 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zv49l_cab85095-f633-4d39-aebd-3716781e6c66/istio-proxy/0.log" Jul 29 14:58:27.080275 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:27.080236 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zv49l_cab85095-f633-4d39-aebd-3716781e6c66/istio-proxy/1.log" Jul 29 14:58:27.166670 ip-10-0-139-155 kubenswrapper[2560]: I0729 14:58:27.166641 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78f77ccbf5-cjvmf_e04e6603-8385-4ffb-aa85-ccae91e7658f/router/0.log"