Jul 24 17:17:23.231999 ip-10-0-130-58 systemd[1]: Starting Kubernetes Kubelet... Jul 24 17:17:23.577261 ip-10-0-130-58 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:17:23.577261 ip-10-0-130-58 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 17:17:23.577261 ip-10-0-130-58 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:17:23.577261 ip-10-0-130-58 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 17:17:23.577261 ip-10-0-130-58 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:17:23.578324 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.578204 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 17:17:23.582511 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582493 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:17:23.582511 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582511 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582515 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582518 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582521 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582523 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582526 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582529 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582532 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582534 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582537 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582540 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582543 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582545 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582554 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582557 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582559 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582562 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582564 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582567 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582570 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:17:23.582584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582572 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582575 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582578 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582580 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582583 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582585 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582588 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582592 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582597 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582600 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582602 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582605 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582608 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582610 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582613 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582615 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582618 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582620 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582623 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582625 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:17:23.583076 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582628 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582631 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582634 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582638 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582642 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582645 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582647 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582650 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582652 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582655 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582658 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582660 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582663 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582666 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582669 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582671 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582674 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582676 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582678 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:17:23.583593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582681 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582684 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582686 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582688 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582691 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582693 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582695 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582698 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582700 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582703 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582705 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582707 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582710 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582712 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582715 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582717 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582720 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582736 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582739 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582741 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:17:23.584109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582744 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582749 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582752 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582755 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582758 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582762 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582764 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582769 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582773 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582775 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582778 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582781 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582784 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582786 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582789 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582792 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582794 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582796 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.582799 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:17:23.584584 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583579 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583586 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583589 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583591 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583594 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583597 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583600 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583603 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583606 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583610 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583613 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583615 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583618 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583620 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583623 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583625 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583627 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583630 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583633 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583636 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:17:23.585055 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583639 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583642 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583644 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583647 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583650 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583653 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583655 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583658 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583660 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583663 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583665 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583670 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583674 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583678 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583682 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583685 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583687 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583690 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583693 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:17:23.585539 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583695 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583698 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583701 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583704 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583707 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583709 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583712 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583714 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583717 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583721 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583741 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583743 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583746 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583749 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583752 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583755 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583757 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583760 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583763 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583766 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:17:23.586056 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583768 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583771 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583773 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583776 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583778 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583781 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583784 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583786 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583788 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583791 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583793 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583796 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583798 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583801 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583803 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583808 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583813 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583816 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583818 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:17:23.586538 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583821 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583823 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583826 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583829 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583831 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583834 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583837 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583839 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583842 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583845 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583847 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583849 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583852 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583854 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583857 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583859 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583862 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583864 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583867 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583869 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:17:23.587017 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.583871 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584250 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584255 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584258 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584262 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584266 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584269 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584271 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584274 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584288 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584291 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584294 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584296 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584299 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584301 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584304 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584307 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584310 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584313 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:17:23.587509 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584315 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584318 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584320 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584323 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584325 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584328 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584330 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584333 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584336 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584338 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584341 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584343 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584345 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584348 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584350 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584353 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584355 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584357 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584360 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584362 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:17:23.588000 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584365 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584368 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584371 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584374 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584376 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584379 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584381 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584384 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584386 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584392 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584394 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584396 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584399 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584401 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584403 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584406 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584408 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584410 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584413 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584415 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:17:23.588487 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584418 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584420 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584422 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584426 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584428 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584430 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584433 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584435 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584438 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584441 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584443 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584446 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584448 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584454 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584456 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584459 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584462 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584464 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584467 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:17:23.588979 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584469 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584473 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584478 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584481 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584484 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584487 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584490 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584493 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584497 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584500 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584502 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584505 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584507 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584510 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584512 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584515 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584517 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584520 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584522 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584525 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:17:23.589431 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584527 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.584529 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584606 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584614 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584621 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584625 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584631 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584634 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584642 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584647 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584652 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584655 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584658 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584661 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584666 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584669 2574 flags.go:64] FLAG: --cgroup-root="" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584672 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584675 2574 flags.go:64] FLAG: --client-ca-file="" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584678 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.584681 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585736 2574 flags.go:64] FLAG: --cluster-domain="" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585740 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585744 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585747 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 17:17:23.589926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585751 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585755 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585758 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585761 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585764 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585767 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585770 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585773 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585776 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585779 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585783 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585786 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585789 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585791 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585796 2574 flags.go:64] FLAG: --enable-server="true" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585799 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585803 2574 flags.go:64] FLAG: --event-burst="100" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585807 2574 flags.go:64] FLAG: --event-qps="50" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585809 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585813 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585816 2574 flags.go:64] FLAG: --eviction-hard="" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585820 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585823 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585826 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585829 2574 flags.go:64] FLAG: --eviction-soft="" Jul 24 17:17:23.590506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585832 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585835 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585838 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585841 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585843 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585846 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585849 2574 flags.go:64] FLAG: --feature-gates="" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585853 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585856 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585859 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585862 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585865 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585868 2574 flags.go:64] FLAG: --help="false" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585871 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-130-58.ec2.internal" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585874 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585877 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585880 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585883 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585887 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585890 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585893 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585897 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585899 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 17:17:23.591133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585902 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585905 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585908 2574 flags.go:64] FLAG: --kube-reserved="" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585911 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585914 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585917 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585920 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585923 2574 flags.go:64] FLAG: --lock-file="" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585925 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585939 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585943 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585949 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585951 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585954 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585957 2574 flags.go:64] FLAG: --logging-format="text" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585960 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585963 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585966 2574 flags.go:64] FLAG: --manifest-url="" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585968 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585973 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585976 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585980 2574 flags.go:64] FLAG: --max-pods="110" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585983 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585986 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585989 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 17:17:23.591702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585991 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585994 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.585997 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586000 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586008 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586011 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586014 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586017 2574 flags.go:64] FLAG: --pod-cidr="" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586020 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586034 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586037 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586040 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586044 2574 flags.go:64] FLAG: --port="10250" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586047 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586050 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03408e8a34df580df" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586053 2574 flags.go:64] FLAG: --qos-reserved="" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586056 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586059 2574 flags.go:64] FLAG: --register-node="true" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586061 2574 flags.go:64] FLAG: --register-with-taints="" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586065 2574 flags.go:64] FLAG: --registry-burst="10" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586068 2574 flags.go:64] FLAG: --registry-qps="5" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586071 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586074 2574 flags.go:64] FLAG: --reserved-memory="" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586078 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586081 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 17:17:23.592390 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586083 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586086 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586089 2574 flags.go:64] FLAG: --runonce="false" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586092 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586095 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586098 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586101 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586104 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586107 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586110 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586113 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586115 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586118 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586121 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586124 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586127 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586130 2574 flags.go:64] FLAG: --system-cgroups="" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586132 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586138 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586140 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586144 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586148 2574 flags.go:64] FLAG: --tls-min-version="" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586150 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586153 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586155 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 17:17:23.593025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586158 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586161 2574 flags.go:64] FLAG: --v="2" Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586165 2574 flags.go:64] FLAG: --version="false" Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586168 2574 flags.go:64] FLAG: --vmodule="" Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586172 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.586175 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586293 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586297 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586300 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586303 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586305 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586308 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586310 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586312 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586315 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586317 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586320 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586322 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586324 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586326 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586329 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:17:23.593620 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586332 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586334 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586337 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586339 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586343 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586345 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586348 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586351 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586354 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586356 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586359 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586361 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586364 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586366 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586369 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586371 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586374 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586377 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586379 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586381 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:17:23.594143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586384 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586386 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586389 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586391 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586393 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586396 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586398 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586401 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586403 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586406 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586408 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586411 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586413 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586415 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586418 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586420 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586424 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586426 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586429 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:17:23.594621 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586431 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586435 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586439 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586441 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586445 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586449 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586452 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586454 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586457 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586459 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586462 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586464 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586467 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586469 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586471 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586474 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586476 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586478 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586481 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:17:23.595232 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586483 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586486 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586488 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586490 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586492 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586495 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586497 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586501 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586504 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586507 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586511 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586514 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586517 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586519 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586522 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586525 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586528 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586530 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586533 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586535 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:17:23.595697 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586538 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586540 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586543 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586546 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586549 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.586552 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.587057 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.593566 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.593581 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.593601 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.593607 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593664 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593669 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593672 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593675 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:17:23.596213 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593679 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593682 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593685 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593687 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593690 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593693 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593695 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593699 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593703 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593706 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593709 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593712 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593715 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593718 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593738 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593743 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593746 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593750 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593755 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:17:23.596598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593758 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593761 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593764 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593766 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593769 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593771 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593774 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593777 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593779 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593782 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593785 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593787 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593790 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593793 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593795 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593797 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593800 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593804 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593808 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593811 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:17:23.597086 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593814 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593817 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593820 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593822 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593825 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593827 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593830 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593833 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593836 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593838 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593841 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593843 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593846 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593848 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593851 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593853 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593856 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593858 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593861 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593863 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:17:23.597568 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593866 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593868 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593872 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593876 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593879 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593882 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593884 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593887 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593889 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593893 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593895 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593897 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593900 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593903 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593905 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593908 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593910 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593913 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593915 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:17:23.598070 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593918 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593921 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593923 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593926 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593928 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593931 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593933 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593936 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593938 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593941 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593943 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593946 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593949 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593951 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593954 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593956 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:17:23.598562 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.593958 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.593963 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594085 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594091 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594094 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594096 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594099 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594102 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594104 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594107 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594109 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594112 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594114 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594117 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594119 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:17:23.599041 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594122 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594124 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594127 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594130 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594132 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594134 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594137 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594139 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594141 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594144 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594147 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594152 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594156 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594158 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594161 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594164 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594166 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594168 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594171 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:17:23.599408 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594173 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594176 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594178 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594180 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594183 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594185 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594187 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594190 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594192 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594195 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594197 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594199 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594201 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594204 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594206 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594208 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594212 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594214 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594216 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594219 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:17:23.599889 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594223 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594226 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594228 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594231 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594234 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594236 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594239 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594241 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594243 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594246 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594249 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594251 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594253 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594256 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594258 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594260 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594263 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594266 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594268 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:17:23.600369 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594271 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594273 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594276 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594278 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594280 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594283 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594285 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594288 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594290 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594293 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594295 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594298 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594300 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594303 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594305 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594307 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594310 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594312 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594315 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594317 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:17:23.600875 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594319 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594322 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594324 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594327 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594329 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594333 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594335 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:23.594338 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.594343 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.594458 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.597120 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.598759 2574 server.go:1019] "Starting client certificate rotation" Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.598863 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 17:17:23.601341 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.598899 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 17:17:23.601659 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.600916 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 17:17:23.604939 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.604919 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 17:17:23.615042 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.615025 2574 log.go:25] "Validated CRI v1 runtime API" Jul 24 17:17:23.616774 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.616761 2574 log.go:25] "Validated CRI v1 image API" Jul 24 17:17:23.617486 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.617463 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 17:17:23.619752 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.619709 2574 fs.go:135] Filesystem UUIDs: map[4be39315-785f-469b-8775-7d4ea6120f15:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8b39adac-260f-41a6-8e41-64c6484ee12d:/dev/nvme0n1p4] Jul 24 17:17:23.619806 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.619752 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 17:17:23.622980 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.622963 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 17:17:23.624536 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.624419 2574 manager.go:216] Machine: {Timestamp:2026-07-24 17:17:23.623273961 +0000 UTC m=+0.288848588 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098770 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec260ccdaf8b189879ef6ffb66910613 SystemUUID:ec260ccd-af8b-1898-79ef-6ffb66910613 BootID:d8595574-463a-4070-a9a7-ed53e0aa6cf8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:76:28:52:23:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:76:28:52:23:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:5e:f8:b1:a1:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 17:17:23.624536 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.624529 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 17:17:23.624649 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.624603 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 17:17:23.627240 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.627217 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 17:17:23.627371 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.627244 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-58.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 17:17:23.627769 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.627759 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 17:17:23.627799 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.627771 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 17:17:23.627799 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.627779 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 17:17:23.628508 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.628498 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 17:17:23.629202 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.629192 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 17:17:23.631027 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.631016 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 24 17:17:23.631165 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.631155 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 17:17:23.631196 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.631190 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 24 17:17:23.631229 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.631203 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 17:17:23.631229 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.631219 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 17:17:23.631229 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.631228 2574 kubelet.go:398] "Adding apiserver pod source" Jul 24 17:17:23.631364 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.631236 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 17:17:23.631584 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.631570 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 17:17:23.631635 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.631588 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 17:17:23.634456 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.634432 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 17:17:23.636473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.636460 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 17:17:23.636516 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.636490 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 17:17:23.638083 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638070 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638087 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638092 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638099 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638108 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638113 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638123 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638130 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638138 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638144 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638152 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 17:17:23.638152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638157 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 17:17:23.638703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638693 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 17:17:23.638703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.638703 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 17:17:23.641943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.641930 2574 server.go:1268] "Started kubelet" Jul 24 17:17:23.642063 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.642039 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 17:17:23.644765 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.642016 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 17:17:23.644943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.644925 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 17:17:23.645572 ip-10-0-130-58 systemd[1]: Started Kubernetes Kubelet. Jul 24 17:17:23.647265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.646351 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 17:17:23.647265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.646707 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-58.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 17:17:23.647265 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.646821 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 17:17:23.647810 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.647782 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 17:17:23.648700 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.648683 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 24 17:17:23.649049 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.649026 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 17:17:23.650449 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.649754 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-58.ec2.internal.18c54951f3184e78 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-58.ec2.internal,UID:ip-10-0-130-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-58.ec2.internal,},FirstTimestamp:2026-07-24 17:17:23.64190476 +0000 UTC m=+0.307479389,LastTimestamp:2026-07-24 17:17:23.64190476 +0000 UTC m=+0.307479389,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-58.ec2.internal,}" Jul 24 17:17:23.650679 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.650662 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d7wgb" Jul 24 17:17:23.654092 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654073 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 17:17:23.654092 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654091 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 17:17:23.654221 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.654150 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:23.654771 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654715 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 17:17:23.654771 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654770 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 17:17:23.654921 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654786 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 17:17:23.654921 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654882 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 24 17:17:23.654921 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654890 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 24 17:17:23.654921 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654898 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 17:17:23.654921 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654911 2574 factory.go:55] Registering systemd factory Jul 24 17:17:23.654921 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.654920 2574 factory.go:223] Registration of the systemd container factory successfully Jul 24 17:17:23.655184 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.655087 2574 factory.go:153] Registering CRI-O factory Jul 24 17:17:23.655184 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.655100 2574 factory.go:223] Registration of the crio container factory successfully Jul 24 17:17:23.655184 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.655117 2574 factory.go:103] Registering Raw factory Jul 24 17:17:23.655184 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.655131 2574 manager.go:1196] Started watching for new ooms in manager Jul 24 17:17:23.655496 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.655484 2574 manager.go:349] Starting recovery of all containers Jul 24 17:17:23.657709 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.657688 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d7wgb" Jul 24 17:17:23.658150 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.658119 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 17:17:23.658242 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.658209 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 17:17:23.662464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.662447 2574 manager.go:354] Recovery completed Jul 24 17:17:23.669374 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.669358 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:17:23.671191 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.671176 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:17:23.671257 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.671203 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:17:23.671257 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.671213 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:17:23.671693 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.671679 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 17:17:23.671693 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.671677 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:17:23.671849 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.671693 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 17:17:23.671928 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.671862 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 24 17:17:23.672450 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.672435 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:17:23.672535 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.672464 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:17:23.672535 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.672477 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:17:23.673189 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.673124 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-58.ec2.internal.18c54951f4d72ef9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-58.ec2.internal,UID:ip-10-0-130-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-58.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-58.ec2.internal,},FirstTimestamp:2026-07-24 17:17:23.671191289 +0000 UTC m=+0.336765917,LastTimestamp:2026-07-24 17:17:23.671191289 +0000 UTC m=+0.336765917,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-58.ec2.internal,}" Jul 24 17:17:23.673378 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.673365 2574 policy_none.go:49] "None policy: Start" Jul 24 17:17:23.673438 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.673382 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 17:17:23.673438 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.673392 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 17:17:23.674272 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.674262 2574 policy_none.go:47] "Start" Jul 24 17:17:23.710438 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.710261 2574 manager.go:337] "Starting Device Plugin manager" Jul 24 17:17:23.710511 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.710463 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 17:17:23.710511 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.710477 2574 server.go:87] "Starting device plugin registration server" Jul 24 17:17:23.710694 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.710682 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 17:17:23.710773 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.710700 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 17:17:23.711272 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.710973 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 17:17:23.711272 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.711188 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 17:17:23.711272 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.711199 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 17:17:23.711483 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.711405 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 17:17:23.711483 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.711449 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:23.793250 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.793221 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 17:17:23.794555 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.794542 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 17:17:23.794609 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.794577 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 17:17:23.794609 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.794596 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 17:17:23.794686 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.794625 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 17:17:23.797148 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.797127 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:17:23.811103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.811085 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:17:23.811808 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.811795 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:17:23.811866 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.811826 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:17:23.811866 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.811840 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:17:23.811866 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.811861 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-58.ec2.internal" Jul 24 17:17:23.817927 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.817912 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-58.ec2.internal" Jul 24 17:17:23.818009 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.817937 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-58.ec2.internal\": node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:23.832245 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.832192 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:23.895627 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.895600 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal"] Jul 24 17:17:23.895753 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.895671 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:17:23.896538 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.896525 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:17:23.896580 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.896553 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:17:23.896580 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.896562 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:17:23.897670 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.897658 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:17:23.897798 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.897784 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" Jul 24 17:17:23.897847 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.897823 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:17:23.898410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.898396 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:17:23.898491 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.898426 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:17:23.898491 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.898440 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:17:23.898947 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.898931 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:17:23.899034 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.898959 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:17:23.899034 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.898970 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:17:23.900055 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.900045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal" Jul 24 17:17:23.900103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.900065 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:17:23.900704 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.900689 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:17:23.900812 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.900718 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:17:23.900812 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.900748 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:17:23.923321 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.923302 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-58.ec2.internal\" not found" node="ip-10-0-130-58.ec2.internal" Jul 24 17:17:23.927548 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.927530 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-58.ec2.internal\" not found" node="ip-10-0-130-58.ec2.internal" Jul 24 17:17:23.932574 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:23.932559 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:23.956073 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.956052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/989b1384b1c0ba2d2c93129000922e75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal\" (UID: \"989b1384b1c0ba2d2c93129000922e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" Jul 24 17:17:23.956139 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.956078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/989b1384b1c0ba2d2c93129000922e75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal\" (UID: \"989b1384b1c0ba2d2c93129000922e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" Jul 24 17:17:23.956139 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:23.956093 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7670db40d2e103b7e1fb032e1e25eefe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-58.ec2.internal\" (UID: \"7670db40d2e103b7e1fb032e1e25eefe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.032682 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:24.032657 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:24.057008 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.056985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/989b1384b1c0ba2d2c93129000922e75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal\" (UID: \"989b1384b1c0ba2d2c93129000922e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.057095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.057027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/989b1384b1c0ba2d2c93129000922e75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal\" (UID: \"989b1384b1c0ba2d2c93129000922e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.057095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.057057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7670db40d2e103b7e1fb032e1e25eefe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-58.ec2.internal\" (UID: \"7670db40d2e103b7e1fb032e1e25eefe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.057095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.057080 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/989b1384b1c0ba2d2c93129000922e75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal\" (UID: \"989b1384b1c0ba2d2c93129000922e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.057095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.057085 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/989b1384b1c0ba2d2c93129000922e75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal\" (UID: \"989b1384b1c0ba2d2c93129000922e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.057233 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.057090 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7670db40d2e103b7e1fb032e1e25eefe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-58.ec2.internal\" (UID: \"7670db40d2e103b7e1fb032e1e25eefe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.133406 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:24.133340 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:24.225944 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.225911 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.230871 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.230854 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.233621 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:24.233597 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:24.334268 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:24.334232 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:24.434861 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:24.434786 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:24.535366 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:24.535320 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:24.597565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.597526 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 17:17:24.598042 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.597701 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 17:17:24.635882 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:24.635851 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-58.ec2.internal\" not found" Jul 24 17:17:24.663052 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.663012 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 17:12:24 +0000 UTC" deadline="2028-02-26 16:09:02.167832199 +0000 UTC" Jul 24 17:17:24.663052 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.663042 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13966h51m37.504793251s" Jul 24 17:17:24.714209 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:24.714167 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989b1384b1c0ba2d2c93129000922e75.slice/crio-5c43dcb4ff7cae292f946f2aeed2c7f8fa10f1ef47579f4b0f56c7c63198ea8f WatchSource:0}: Error finding container 5c43dcb4ff7cae292f946f2aeed2c7f8fa10f1ef47579f4b0f56c7c63198ea8f: Status 404 returned error can't find the container with id 5c43dcb4ff7cae292f946f2aeed2c7f8fa10f1ef47579f4b0f56c7c63198ea8f Jul 24 17:17:24.718574 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.718554 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:17:24.732074 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.732048 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:17:24.755291 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.755264 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.773075 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.773053 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 17:17:24.775047 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.775030 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal" Jul 24 17:17:24.781043 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.780629 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:17:24.787639 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.787623 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 17:17:24.798254 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:24.798212 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" event={"ID":"989b1384b1c0ba2d2c93129000922e75","Type":"ContainerStarted","Data":"5c43dcb4ff7cae292f946f2aeed2c7f8fa10f1ef47579f4b0f56c7c63198ea8f"} Jul 24 17:17:24.856749 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:24.856710 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7670db40d2e103b7e1fb032e1e25eefe.slice/crio-a8ca8621bac9e810461c433782fe0ef1aa2732caedec3b1c376bb3190d0adaf2 WatchSource:0}: Error finding container a8ca8621bac9e810461c433782fe0ef1aa2732caedec3b1c376bb3190d0adaf2: Status 404 returned error can't find the container with id a8ca8621bac9e810461c433782fe0ef1aa2732caedec3b1c376bb3190d0adaf2 Jul 24 17:17:25.107984 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.107907 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:17:25.631770 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.631719 2574 apiserver.go:52] "Watching apiserver" Jul 24 17:17:25.641471 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.641439 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 17:17:25.642975 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.642946 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-z5h8g","kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal","openshift-cluster-node-tuning-operator/tuned-hjt6l","openshift-image-registry/node-ca-c4n6l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal","openshift-multus/multus-additional-cni-plugins-mb279","openshift-multus/multus-dxrt4","openshift-multus/network-metrics-daemon-6t9nt","kube-system/global-pull-secret-syncer-jm92j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5","openshift-dns/node-resolver-9t9vf","openshift-network-diagnostics/network-check-target-z2lgx","openshift-network-operator/iptables-alerter-rsvkh","openshift-ovn-kubernetes/ovnkube-node-qnk2q"] Jul 24 17:17:25.644616 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.644595 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.646363 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.646342 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.648848 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.648787 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.648954 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.648874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.649581 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.649561 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 17:17:25.649731 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.649565 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 17:17:25.649731 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.649680 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 17:17:25.649949 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.649888 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vq2kj\"" Jul 24 17:17:25.649949 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.649913 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 17:17:25.650063 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.650008 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.651848 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.651301 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.651848 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.651837 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 17:17:25.652009 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.651879 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 17:17:25.652009 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.651932 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-94ffq\"" Jul 24 17:17:25.652009 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.651885 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:17:25.652642 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.652625 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 17:17:25.652998 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.652979 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 17:17:25.653660 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.653641 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 17:17:25.653901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.653883 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bx54w\"" Jul 24 17:17:25.654217 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.654200 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 17:17:25.654846 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.654827 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j87t6\"" Jul 24 17:17:25.655295 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.655261 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bcqhl\"" Jul 24 17:17:25.655505 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.655487 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 17:17:25.655756 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.655734 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 17:17:25.656216 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.656195 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 17:17:25.656403 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.656383 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 17:17:25.656515 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.656496 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 17:17:25.656862 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.656841 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 17:17:25.657151 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.657130 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 17:17:25.657261 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.657244 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 17:17:25.657321 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.657280 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mcqph\"" Jul 24 17:17:25.657379 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.657358 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 17:17:25.661330 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.661283 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:25.661424 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.661376 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:25.668103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668079 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.668594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzxj7\" (UniqueName: \"kubernetes.io/projected/04f63ec9-fb88-475f-b6f2-89ccec28afe9-kube-api-access-vzxj7\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.668677 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-host\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.668677 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5cf\" (UniqueName: \"kubernetes.io/projected/f74c8892-223e-4771-ab9f-34954297b06d-kube-api-access-qj5cf\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.668677 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-kubelet\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.668865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f16ccb7-8012-4838-9b38-90dabf3e05b0-env-overrides\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.668865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668707 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:25.668865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-run-k8s-cni-cncf-io\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.668865 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.668794 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:25.668865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-var-lib-kubelet\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.669101 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c3a820a-6416-462c-a681-f0aa4da52e8c-host\") pod \"node-ca-c4n6l\" (UID: \"4c3a820a-6416-462c-a681-f0aa4da52e8c\") " pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.669101 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-node-log\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.669101 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8ht\" (UniqueName: \"kubernetes.io/projected/0f16ccb7-8012-4838-9b38-90dabf3e05b0-kube-api-access-6g8ht\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.669101 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.668998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-system-cni-dir\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.669101 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-system-cni-dir\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.669297 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91-hosts-file\") pod \"node-resolver-9t9vf\" (UID: \"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91\") " pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.669354 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjbk\" (UniqueName: \"kubernetes.io/projected/6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91-kube-api-access-nzjbk\") pod \"node-resolver-9t9vf\" (UID: \"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91\") " pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.669400 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-sysconfig\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.669514 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-kubernetes\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.669574 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-lib-modules\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.669574 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-run-openvswitch\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.669671 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-conf-dir\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.669727 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-sysctl-conf\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.669727 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-run-netns\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.669848 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f16ccb7-8012-4838-9b38-90dabf3e05b0-ovnkube-config\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.669892 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f16ccb7-8012-4838-9b38-90dabf3e05b0-ovn-node-metrics-cert\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.669929 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:25.669975 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-run-multus-certs\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.669975 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.670042 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.669979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-sys\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.670042 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-slash\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.670104 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-var-lib-cni-multus\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.670104 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670071 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-systemd\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.670104 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-log-socket\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.670198 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670118 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-os-release\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.670198 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmm7\" (UniqueName: \"kubernetes.io/projected/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-kube-api-access-dwmm7\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.670198 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-var-lib-kubelet\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.670198 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.670319 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.670319 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670249 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2dhm\" (UniqueName: \"kubernetes.io/projected/4c3a820a-6416-462c-a681-f0aa4da52e8c-kube-api-access-k2dhm\") pod \"node-ca-c4n6l\" (UID: \"4c3a820a-6416-462c-a681-f0aa4da52e8c\") " pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.670319 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a93307e-c6da-4166-861e-74ad20b421b0-kubelet-config\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:25.670410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f74c8892-223e-4771-ab9f-34954297b06d-etc-tuned\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.670410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670359 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-run-systemd\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.670410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-var-lib-openvswitch\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.670504 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670411 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f16ccb7-8012-4838-9b38-90dabf3e05b0-ovnkube-script-lib\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.670504 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670480 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-var-lib-cni-bin\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.670565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-hostroot\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.670619 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.670685 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-cnibin\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.670733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-run-ovn\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.670788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f74c8892-223e-4771-ab9f-34954297b06d-tmp\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.670861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c3a820a-6416-462c-a681-f0aa4da52e8c-serviceca\") pod \"node-ca-c4n6l\" (UID: \"4c3a820a-6416-462c-a681-f0aa4da52e8c\") " pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.670861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.670943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-cni-netd\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.670943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.670943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670913 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04f63ec9-fb88-475f-b6f2-89ccec28afe9-cni-binary-copy\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.671103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-run-netns\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.671103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.670990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-cnibin\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.671103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-os-release\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.671103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671050 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91-tmp-dir\") pod \"node-resolver-9t9vf\" (UID: \"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91\") " pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.671103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671087 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-sysctl-d\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.671299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-run\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.671299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-systemd-units\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.671299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-cni-dir\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.671299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-modprobe-d\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.671299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671261 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-etc-openvswitch\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.671299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-cni-bin\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.671583 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a93307e-c6da-4166-861e-74ad20b421b0-dbus\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:25.671583 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-socket-dir-parent\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.671583 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-daemon-config\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.671583 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-etc-kubernetes\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.671787 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671688 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:17:25.671787 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.671698 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.672314 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.672297 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 17:17:25.672314 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.672310 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 17:17:25.672938 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.672740 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-599f8\"" Jul 24 17:17:25.673966 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.673946 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:25.674420 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.674344 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 17:17:25.676764 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.675192 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 17:17:25.676764 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.675568 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8l7jr\"" Jul 24 17:17:25.676764 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.675773 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:25.676764 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.675823 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 17:17:25.676764 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.675834 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:25.676764 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.675940 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 17:17:25.677553 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.677534 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 17:17:25.678078 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.677900 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 17:17:25.678078 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.678049 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-98grh\"" Jul 24 17:17:25.696152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.696133 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5dzm9" Jul 24 17:17:25.703867 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.703844 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5dzm9" Jul 24 17:17:25.756126 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.756099 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-var-lib-cni-multus\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-systemd\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-log-socket\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-socket-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771689 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67675beb-50d9-4473-9216-82ec5f9b78a5-iptables-alerter-script\") pod \"iptables-alerter-rsvkh\" (UID: \"67675beb-50d9-4473-9216-82ec5f9b78a5\") " pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-os-release\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771727 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmm7\" (UniqueName: \"kubernetes.io/projected/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-kube-api-access-dwmm7\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771728 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-systemd\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-var-lib-kubelet\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771804 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2dhm\" (UniqueName: \"kubernetes.io/projected/4c3a820a-6416-462c-a681-f0aa4da52e8c-kube-api-access-k2dhm\") pod \"node-ca-c4n6l\" (UID: \"4c3a820a-6416-462c-a681-f0aa4da52e8c\") " pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-log-socket\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a93307e-c6da-4166-861e-74ad20b421b0-kubelet-config\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-var-lib-cni-multus\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.775421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.771987 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-var-lib-kubelet\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f74c8892-223e-4771-ab9f-34954297b06d-etc-tuned\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-run-systemd\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-var-lib-openvswitch\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f16ccb7-8012-4838-9b38-90dabf3e05b0-ovnkube-script-lib\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772165 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-etc-selinux\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-var-lib-cni-bin\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-hostroot\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31-konnectivity-ca\") pod \"konnectivity-agent-z5h8g\" (UID: \"4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31\") " pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-sys-fs\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-cnibin\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-run-ovn\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f74c8892-223e-4771-ab9f-34954297b06d-tmp\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c3a820a-6416-462c-a681-f0aa4da52e8c-serviceca\") pod \"node-ca-c4n6l\" (UID: \"4c3a820a-6416-462c-a681-f0aa4da52e8c\") " pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-cni-netd\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-registration-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67675beb-50d9-4473-9216-82ec5f9b78a5-host-slash\") pod \"iptables-alerter-rsvkh\" (UID: \"67675beb-50d9-4473-9216-82ec5f9b78a5\") " pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04f63ec9-fb88-475f-b6f2-89ccec28afe9-cni-binary-copy\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-run-netns\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-cnibin\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772490 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772538 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-run-systemd\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-var-lib-openvswitch\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772591 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-os-release\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772647 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a93307e-c6da-4166-861e-74ad20b421b0-kubelet-config\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772953 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-os-release\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773074 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f16ccb7-8012-4838-9b38-90dabf3e05b0-ovnkube-script-lib\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773287 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-cni-netd\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.776788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773300 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.772497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-os-release\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773350 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91-tmp-dir\") pod \"node-resolver-9t9vf\" (UID: \"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91\") " pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-sysctl-d\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-run\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-systemd-units\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c3a820a-6416-462c-a681-f0aa4da52e8c-serviceca\") pod \"node-ca-c4n6l\" (UID: \"4c3a820a-6416-462c-a681-f0aa4da52e8c\") " pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-cni-dir\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-modprobe-d\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-etc-openvswitch\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-cni-bin\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a93307e-c6da-4166-861e-74ad20b421b0-dbus\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773600 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzpl6\" (UniqueName: \"kubernetes.io/projected/e4e8e0d2-7d16-49b4-86af-9556d854586d-kube-api-access-fzpl6\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-socket-dir-parent\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-daemon-config\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-etc-kubernetes\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777366 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-device-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzxj7\" (UniqueName: \"kubernetes.io/projected/04f63ec9-fb88-475f-b6f2-89ccec28afe9-kube-api-access-vzxj7\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-host\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773830 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5cf\" (UniqueName: \"kubernetes.io/projected/f74c8892-223e-4771-ab9f-34954297b06d-kube-api-access-qj5cf\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773828 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-kubelet\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f16ccb7-8012-4838-9b38-90dabf3e05b0-env-overrides\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.773961 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04f63ec9-fb88-475f-b6f2-89ccec28afe9-cni-binary-copy\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-run-netns\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774020 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-var-lib-cni-bin\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774049 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-cnibin\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-hostroot\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31-agent-certs\") pod \"konnectivity-agent-z5h8g\" (UID: \"4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31\") " pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-run-k8s-cni-cncf-io\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-var-lib-kubelet\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a93307e-c6da-4166-861e-74ad20b421b0-dbus\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c3a820a-6416-462c-a681-f0aa4da52e8c-host\") pod \"node-ca-c4n6l\" (UID: \"4c3a820a-6416-462c-a681-f0aa4da52e8c\") " pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.777926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-run\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-node-log\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-cni-bin\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-systemd-units\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8ht\" (UniqueName: \"kubernetes.io/projected/0f16ccb7-8012-4838-9b38-90dabf3e05b0-kube-api-access-6g8ht\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774265 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-cnibin\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-etc-kubernetes\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-system-cni-dir\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-system-cni-dir\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774630 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f16ccb7-8012-4838-9b38-90dabf3e05b0-env-overrides\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91-hosts-file\") pod \"node-resolver-9t9vf\" (UID: \"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91\") " pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjbk\" (UniqueName: \"kubernetes.io/projected/6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91-kube-api-access-nzjbk\") pod \"node-resolver-9t9vf\" (UID: \"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91\") " pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-sysconfig\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-kubernetes\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-lib-modules\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-run-openvswitch\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-run-openvswitch\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-daemon-config\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.778473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774911 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91-tmp-dir\") pod \"node-resolver-9t9vf\" (UID: \"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91\") " pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774901 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-socket-dir-parent\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774913 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-modprobe-d\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774932 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-run-k8s-cni-cncf-io\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-conf-dir\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774984 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-system-cni-dir\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-sysctl-conf\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775005 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91-hosts-file\") pod \"node-resolver-9t9vf\" (UID: \"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91\") " pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775024 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c3a820a-6416-462c-a681-f0aa4da52e8c-host\") pod \"node-ca-c4n6l\" (UID: \"4c3a820a-6416-462c-a681-f0aa4da52e8c\") " pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-run-netns\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-var-lib-kubelet\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f16ccb7-8012-4838-9b38-90dabf3e05b0-ovnkube-config\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f16ccb7-8012-4838-9b38-90dabf3e05b0-ovn-node-metrics-cert\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccgf7\" (UniqueName: \"kubernetes.io/projected/67675beb-50d9-4473-9216-82ec5f9b78a5-kube-api-access-ccgf7\") pod \"iptables-alerter-rsvkh\" (UID: \"67675beb-50d9-4473-9216-82ec5f9b78a5\") " pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-sysconfig\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-run-multus-certs\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.779079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775271 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-kubernetes\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-sys\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-slash\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.774975 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-cni-dir\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-lib-modules\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775465 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-sys\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775488 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f16ccb7-8012-4838-9b38-90dabf3e05b0-ovnkube-config\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775501 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-etc-openvswitch\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775515 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-host\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-system-cni-dir\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7ln\" (UniqueName: \"kubernetes.io/projected/bd39d23d-0f58-49d6-8d42-39caa7c31c41-kube-api-access-7c7ln\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775562 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-kubelet\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775581 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-slash\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775622 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-multus-conf-dir\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.775676 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-sysctl-conf\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.775852 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret podName:6a93307e-c6da-4166-861e-74ad20b421b0 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:26.275826601 +0000 UTC m=+2.941401234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret") pod "global-pull-secret-syncer-jm92j" (UID: "6a93307e-c6da-4166-861e-74ad20b421b0") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:25.779733 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775852 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-host-run-netns\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.780865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775879 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-run-ovn\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.780865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.775900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04f63ec9-fb88-475f-b6f2-89ccec28afe9-host-run-multus-certs\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.780865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.776052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.780865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.776085 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f16ccb7-8012-4838-9b38-90dabf3e05b0-node-log\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.780865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.776116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f74c8892-223e-4771-ab9f-34954297b06d-etc-sysctl-d\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.783981 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.783951 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8ht\" (UniqueName: \"kubernetes.io/projected/0f16ccb7-8012-4838-9b38-90dabf3e05b0-kube-api-access-6g8ht\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.784423 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.784399 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f74c8892-223e-4771-ab9f-34954297b06d-tmp\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.786348 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.786328 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f16ccb7-8012-4838-9b38-90dabf3e05b0-ovn-node-metrics-cert\") pod \"ovnkube-node-qnk2q\" (UID: \"0f16ccb7-8012-4838-9b38-90dabf3e05b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:25.787110 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.787091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmm7\" (UniqueName: \"kubernetes.io/projected/5fc258ae-e5cd-4f16-84bf-cd2baf654e1a-kube-api-access-dwmm7\") pod \"multus-additional-cni-plugins-mb279\" (UID: \"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a\") " pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:25.787448 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.787423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2dhm\" (UniqueName: \"kubernetes.io/projected/4c3a820a-6416-462c-a681-f0aa4da52e8c-kube-api-access-k2dhm\") pod \"node-ca-c4n6l\" (UID: \"4c3a820a-6416-462c-a681-f0aa4da52e8c\") " pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.789595 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.789569 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjbk\" (UniqueName: \"kubernetes.io/projected/6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91-kube-api-access-nzjbk\") pod \"node-resolver-9t9vf\" (UID: \"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91\") " pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.790271 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.790232 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5cf\" (UniqueName: \"kubernetes.io/projected/f74c8892-223e-4771-ab9f-34954297b06d-kube-api-access-qj5cf\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.791939 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.791918 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f74c8892-223e-4771-ab9f-34954297b06d-etc-tuned\") pod \"tuned-hjt6l\" (UID: \"f74c8892-223e-4771-ab9f-34954297b06d\") " pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.793129 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.793109 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzxj7\" (UniqueName: \"kubernetes.io/projected/04f63ec9-fb88-475f-b6f2-89ccec28afe9-kube-api-access-vzxj7\") pod \"multus-dxrt4\" (UID: \"04f63ec9-fb88-475f-b6f2-89ccec28afe9\") " pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.801899 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.801871 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal" event={"ID":"7670db40d2e103b7e1fb032e1e25eefe","Type":"ContainerStarted","Data":"a8ca8621bac9e810461c433782fe0ef1aa2732caedec3b1c376bb3190d0adaf2"} Jul 24 17:17:25.876122 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-registration-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876122 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67675beb-50d9-4473-9216-82ec5f9b78a5-host-slash\") pod \"iptables-alerter-rsvkh\" (UID: \"67675beb-50d9-4473-9216-82ec5f9b78a5\") " pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.876355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpl6\" (UniqueName: \"kubernetes.io/projected/e4e8e0d2-7d16-49b4-86af-9556d854586d-kube-api-access-fzpl6\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-registration-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-device-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876254 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-device-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876252 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67675beb-50d9-4473-9216-82ec5f9b78a5-host-slash\") pod \"iptables-alerter-rsvkh\" (UID: \"67675beb-50d9-4473-9216-82ec5f9b78a5\") " pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.876355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:25.876355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31-agent-certs\") pod \"konnectivity-agent-z5h8g\" (UID: \"4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31\") " pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:25.876355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccgf7\" (UniqueName: \"kubernetes.io/projected/67675beb-50d9-4473-9216-82ec5f9b78a5-kube-api-access-ccgf7\") pod \"iptables-alerter-rsvkh\" (UID: \"67675beb-50d9-4473-9216-82ec5f9b78a5\") " pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7ln\" (UniqueName: \"kubernetes.io/projected/bd39d23d-0f58-49d6-8d42-39caa7c31c41-kube-api-access-7c7ln\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-socket-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67675beb-50d9-4473-9216-82ec5f9b78a5-iptables-alerter-script\") pod \"iptables-alerter-rsvkh\" (UID: \"67675beb-50d9-4473-9216-82ec5f9b78a5\") " pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.876439 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-etc-selinux\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.876500 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:26.376478336 +0000 UTC m=+3.042052955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31-konnectivity-ca\") pod \"konnectivity-agent-z5h8g\" (UID: \"4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31\") " pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876568 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-sys-fs\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-sys-fs\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.876861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.876769 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-socket-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.877329 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.877091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.877329 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.877211 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e4e8e0d2-7d16-49b4-86af-9556d854586d-etc-selinux\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.877329 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.877307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67675beb-50d9-4473-9216-82ec5f9b78a5-iptables-alerter-script\") pod \"iptables-alerter-rsvkh\" (UID: \"67675beb-50d9-4473-9216-82ec5f9b78a5\") " pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.877831 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.877782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31-konnectivity-ca\") pod \"konnectivity-agent-z5h8g\" (UID: \"4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31\") " pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:25.879905 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.879882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31-agent-certs\") pod \"konnectivity-agent-z5h8g\" (UID: \"4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31\") " pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:25.885666 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.885611 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:17:25.885666 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.885631 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:17:25.885666 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.885644 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tcgdc for pod openshift-network-diagnostics/network-check-target-z2lgx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:25.885917 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:25.885719 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc podName:fe26a767-49a6-4251-be25-4b62a0b0a494 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:26.385698562 +0000 UTC m=+3.051273201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tcgdc" (UniqueName: "kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc") pod "network-check-target-z2lgx" (UID: "fe26a767-49a6-4251-be25-4b62a0b0a494") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:25.888291 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.888265 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7ln\" (UniqueName: \"kubernetes.io/projected/bd39d23d-0f58-49d6-8d42-39caa7c31c41-kube-api-access-7c7ln\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:25.889083 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.889051 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccgf7\" (UniqueName: \"kubernetes.io/projected/67675beb-50d9-4473-9216-82ec5f9b78a5-kube-api-access-ccgf7\") pod \"iptables-alerter-rsvkh\" (UID: \"67675beb-50d9-4473-9216-82ec5f9b78a5\") " pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:25.889083 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.889063 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpl6\" (UniqueName: \"kubernetes.io/projected/e4e8e0d2-7d16-49b4-86af-9556d854586d-kube-api-access-fzpl6\") pod \"aws-ebs-csi-driver-node-txbs5\" (UID: \"e4e8e0d2-7d16-49b4-86af-9556d854586d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:25.959267 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.959228 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxrt4" Jul 24 17:17:25.972981 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.972954 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9t9vf" Jul 24 17:17:25.982625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.982600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" Jul 24 17:17:25.991499 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.991477 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4n6l" Jul 24 17:17:25.998333 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:25.998314 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:26.006885 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.006865 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mb279" Jul 24 17:17:26.016706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.016688 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rsvkh" Jul 24 17:17:26.025572 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.025543 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" Jul 24 17:17:26.032463 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.032444 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:26.183035 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.182940 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:17:26.284138 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.284094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:26.284313 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:26.284233 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:26.284313 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:26.284310 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret podName:6a93307e-c6da-4166-861e-74ad20b421b0 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:27.28428943 +0000 UTC m=+3.949864068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret") pod "global-pull-secret-syncer-jm92j" (UID: "6a93307e-c6da-4166-861e-74ad20b421b0") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:26.384503 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.384461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:26.384666 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:26.384592 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:26.384666 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:26.384652 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:27.384637865 +0000 UTC m=+4.050212483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:26.485310 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.485211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:26.485492 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:26.485422 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:17:26.485492 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:26.485450 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:17:26.485492 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:26.485463 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tcgdc for pod openshift-network-diagnostics/network-check-target-z2lgx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:26.485645 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:26.485535 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc podName:fe26a767-49a6-4251-be25-4b62a0b0a494 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:27.485514464 +0000 UTC m=+4.151089083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcgdc" (UniqueName: "kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc") pod "network-check-target-z2lgx" (UID: "fe26a767-49a6-4251-be25-4b62a0b0a494") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:26.705933 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.705876 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:12:26 +0000 UTC" deadline="2028-01-03 21:29:42.021695541 +0000 UTC" Jul 24 17:17:26.705933 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.705923 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12676h12m15.315775105s" Jul 24 17:17:26.794908 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:26.794828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:26.795057 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:26.794952 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:26.937273 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:26.937242 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f16ccb7_8012_4838_9b38_90dabf3e05b0.slice/crio-e84964f89ba29a2f9935ff76022c9f03744afa9824bf2c322c9e60b5ddd24b54 WatchSource:0}: Error finding container e84964f89ba29a2f9935ff76022c9f03744afa9824bf2c322c9e60b5ddd24b54: Status 404 returned error can't find the container with id e84964f89ba29a2f9935ff76022c9f03744afa9824bf2c322c9e60b5ddd24b54 Jul 24 17:17:26.938203 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:26.938162 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3a820a_6416_462c_a681_f0aa4da52e8c.slice/crio-5fcecacd4466a2465e84dcc20cadd6c9d047a502a02097459596ff13f0b008be WatchSource:0}: Error finding container 5fcecacd4466a2465e84dcc20cadd6c9d047a502a02097459596ff13f0b008be: Status 404 returned error can't find the container with id 5fcecacd4466a2465e84dcc20cadd6c9d047a502a02097459596ff13f0b008be Jul 24 17:17:26.954982 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:26.954893 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc9ee97_5c1b_428b_ba1e_eca9eccd2e91.slice/crio-f5f90b6d8ba2b1b006081562b435a3078039127f65dae238dc434272843d5f02 WatchSource:0}: Error finding container f5f90b6d8ba2b1b006081562b435a3078039127f65dae238dc434272843d5f02: Status 404 returned error can't find the container with id f5f90b6d8ba2b1b006081562b435a3078039127f65dae238dc434272843d5f02 Jul 24 17:17:26.973212 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:26.973184 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2e73a1_f7cf_48bc_b4f5_15bb101e0a31.slice/crio-ef0d20de0731a4e31dfb07112a402ded8fee9bc0fa4043a75d1033cf2761fa08 WatchSource:0}: Error finding container ef0d20de0731a4e31dfb07112a402ded8fee9bc0fa4043a75d1033cf2761fa08: Status 404 returned error can't find the container with id ef0d20de0731a4e31dfb07112a402ded8fee9bc0fa4043a75d1033cf2761fa08 Jul 24 17:17:26.974547 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:26.974522 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67675beb_50d9_4473_9216_82ec5f9b78a5.slice/crio-86f23f605f5b908757e110f139b8c39ff1415b5488b15dd7b14d700d6fca66ff WatchSource:0}: Error finding container 86f23f605f5b908757e110f139b8c39ff1415b5488b15dd7b14d700d6fca66ff: Status 404 returned error can't find the container with id 86f23f605f5b908757e110f139b8c39ff1415b5488b15dd7b14d700d6fca66ff Jul 24 17:17:26.975386 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:26.975357 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e8e0d2_7d16_49b4_86af_9556d854586d.slice/crio-ee12b52ab66c58d75190b58e0b62017e876a441e98c48d9a91c2b332923d5325 WatchSource:0}: Error finding container ee12b52ab66c58d75190b58e0b62017e876a441e98c48d9a91c2b332923d5325: Status 404 returned error can't find the container with id ee12b52ab66c58d75190b58e0b62017e876a441e98c48d9a91c2b332923d5325 Jul 24 17:17:26.976389 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:26.976345 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74c8892_223e_4771_ab9f_34954297b06d.slice/crio-46dbc0a986a469a2990f24141230d007b778eb3f1d9893bc8a648cf3b1938ef0 WatchSource:0}: Error finding container 46dbc0a986a469a2990f24141230d007b778eb3f1d9893bc8a648cf3b1938ef0: Status 404 returned error can't find the container with id 46dbc0a986a469a2990f24141230d007b778eb3f1d9893bc8a648cf3b1938ef0 Jul 24 17:17:26.977626 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:26.977606 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f63ec9_fb88_475f_b6f2_89ccec28afe9.slice/crio-8c1b7c577491dc8d5c4f5a2f3d65d961ede343aa0169f584124fd6e5fd69e033 WatchSource:0}: Error finding container 8c1b7c577491dc8d5c4f5a2f3d65d961ede343aa0169f584124fd6e5fd69e033: Status 404 returned error can't find the container with id 8c1b7c577491dc8d5c4f5a2f3d65d961ede343aa0169f584124fd6e5fd69e033 Jul 24 17:17:26.979249 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:17:26.979226 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc258ae_e5cd_4f16_84bf_cd2baf654e1a.slice/crio-df0a33f04d32939a54f534d29100f71a86a52d57fc36d092bfd3869ae399d710 WatchSource:0}: Error finding container df0a33f04d32939a54f534d29100f71a86a52d57fc36d092bfd3869ae399d710: Status 404 returned error can't find the container with id df0a33f04d32939a54f534d29100f71a86a52d57fc36d092bfd3869ae399d710 Jul 24 17:17:27.292167 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.291964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:27.292312 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.292125 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:27.292312 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.292276 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret podName:6a93307e-c6da-4166-861e-74ad20b421b0 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:29.292255044 +0000 UTC m=+5.957829666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret") pod "global-pull-secret-syncer-jm92j" (UID: "6a93307e-c6da-4166-861e-74ad20b421b0") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:27.393174 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.393056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:27.393328 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.393204 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:27.393328 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.393278 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:29.393257813 +0000 UTC m=+6.058832434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:27.494290 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.494247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:27.494489 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.494446 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:17:27.494489 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.494473 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:17:27.494489 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.494485 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tcgdc for pod openshift-network-diagnostics/network-check-target-z2lgx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:27.494643 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.494543 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc podName:fe26a767-49a6-4251-be25-4b62a0b0a494 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:29.494526238 +0000 UTC m=+6.160100855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcgdc" (UniqueName: "kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc") pod "network-check-target-z2lgx" (UID: "fe26a767-49a6-4251-be25-4b62a0b0a494") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:27.706970 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.706865 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:12:26 +0000 UTC" deadline="2028-05-02 06:04:13.09602807 +0000 UTC" Jul 24 17:17:27.706970 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.706912 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15540h46m45.389121108s" Jul 24 17:17:27.796999 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.795939 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:27.796999 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.796106 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:27.796999 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.796505 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:27.796999 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:27.796606 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:27.841126 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.840207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal" event={"ID":"7670db40d2e103b7e1fb032e1e25eefe","Type":"ContainerStarted","Data":"bb80e520cf526a6dfe490e701e59be999c18e271f33d0f86d0a1071b68794151"} Jul 24 17:17:27.848878 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.848717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mb279" event={"ID":"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a","Type":"ContainerStarted","Data":"df0a33f04d32939a54f534d29100f71a86a52d57fc36d092bfd3869ae399d710"} Jul 24 17:17:27.868197 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.868144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z5h8g" event={"ID":"4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31","Type":"ContainerStarted","Data":"ef0d20de0731a4e31dfb07112a402ded8fee9bc0fa4043a75d1033cf2761fa08"} Jul 24 17:17:27.886519 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.886478 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rsvkh" event={"ID":"67675beb-50d9-4473-9216-82ec5f9b78a5","Type":"ContainerStarted","Data":"86f23f605f5b908757e110f139b8c39ff1415b5488b15dd7b14d700d6fca66ff"} Jul 24 17:17:27.899235 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.899190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9t9vf" event={"ID":"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91","Type":"ContainerStarted","Data":"f5f90b6d8ba2b1b006081562b435a3078039127f65dae238dc434272843d5f02"} Jul 24 17:17:27.917198 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.917060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4n6l" event={"ID":"4c3a820a-6416-462c-a681-f0aa4da52e8c","Type":"ContainerStarted","Data":"5fcecacd4466a2465e84dcc20cadd6c9d047a502a02097459596ff13f0b008be"} Jul 24 17:17:27.922768 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.922656 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" event={"ID":"0f16ccb7-8012-4838-9b38-90dabf3e05b0","Type":"ContainerStarted","Data":"e84964f89ba29a2f9935ff76022c9f03744afa9824bf2c322c9e60b5ddd24b54"} Jul 24 17:17:27.954871 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.954676 2574 generic.go:356] "Generic (PLEG): container finished" podID="989b1384b1c0ba2d2c93129000922e75" containerID="096acd534c0260012e50187d003dcd89509193acbb7a005f3cae391343673fa0" exitCode=0 Jul 24 17:17:27.955148 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.955078 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" event={"ID":"989b1384b1c0ba2d2c93129000922e75","Type":"ContainerDied","Data":"096acd534c0260012e50187d003dcd89509193acbb7a005f3cae391343673fa0"} Jul 24 17:17:27.967046 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.967014 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxrt4" event={"ID":"04f63ec9-fb88-475f-b6f2-89ccec28afe9","Type":"ContainerStarted","Data":"8c1b7c577491dc8d5c4f5a2f3d65d961ede343aa0169f584124fd6e5fd69e033"} Jul 24 17:17:27.972418 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.972358 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-58.ec2.internal" podStartSLOduration=2.972338053 podStartE2EDuration="2.972338053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:27.860774442 +0000 UTC m=+4.526349083" watchObservedRunningTime="2026-07-24 17:17:27.972338053 +0000 UTC m=+4.637912695" Jul 24 17:17:27.979682 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.979653 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" event={"ID":"f74c8892-223e-4771-ab9f-34954297b06d","Type":"ContainerStarted","Data":"46dbc0a986a469a2990f24141230d007b778eb3f1d9893bc8a648cf3b1938ef0"} Jul 24 17:17:27.993985 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:27.993890 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" event={"ID":"e4e8e0d2-7d16-49b4-86af-9556d854586d","Type":"ContainerStarted","Data":"ee12b52ab66c58d75190b58e0b62017e876a441e98c48d9a91c2b332923d5325"} Jul 24 17:17:28.795236 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:28.795095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:28.795773 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:28.795234 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:29.011931 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:29.011874 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" event={"ID":"989b1384b1c0ba2d2c93129000922e75","Type":"ContainerStarted","Data":"58bd05b7e405caf8d78d6643c3ac323cf7ca2d35679b2306fa3f90fd8ce45ea4"} Jul 24 17:17:29.313846 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:29.313379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:29.313846 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.313555 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:29.313846 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.313625 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret podName:6a93307e-c6da-4166-861e-74ad20b421b0 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:33.313604722 +0000 UTC m=+9.979179342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret") pod "global-pull-secret-syncer-jm92j" (UID: "6a93307e-c6da-4166-861e-74ad20b421b0") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:29.414706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:29.414671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:29.414913 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.414883 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:29.414988 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.414949 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:33.41492898 +0000 UTC m=+10.080503600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:29.515977 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:29.515835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:29.516181 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.516060 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:17:29.516181 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.516084 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:17:29.516181 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.516096 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tcgdc for pod openshift-network-diagnostics/network-check-target-z2lgx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:29.516181 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.516155 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc podName:fe26a767-49a6-4251-be25-4b62a0b0a494 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:33.516136233 +0000 UTC m=+10.181710853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcgdc" (UniqueName: "kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc") pod "network-check-target-z2lgx" (UID: "fe26a767-49a6-4251-be25-4b62a0b0a494") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:29.795202 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:29.795117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:29.795380 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.795256 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:29.795773 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:29.795684 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:29.796061 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:29.795821 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:30.795756 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:30.795202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:30.795756 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:30.795346 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:31.796444 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:31.795909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:31.796444 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:31.796047 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:31.797011 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:31.796566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:31.797011 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:31.796732 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:32.795679 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:32.795634 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:32.795879 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:32.795807 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:33.349117 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:33.349079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:33.349603 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.349264 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:33.349603 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.349337 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret podName:6a93307e-c6da-4166-861e-74ad20b421b0 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:41.349314999 +0000 UTC m=+18.014889623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret") pod "global-pull-secret-syncer-jm92j" (UID: "6a93307e-c6da-4166-861e-74ad20b421b0") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:33.450041 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:33.449896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:33.450041 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.450028 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:33.450302 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.450114 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:41.450091207 +0000 UTC m=+18.115665827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:33.550932 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:33.550840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:33.551130 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.551071 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:17:33.551130 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.551099 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:17:33.551130 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.551115 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tcgdc for pod openshift-network-diagnostics/network-check-target-z2lgx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:33.551264 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.551182 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc podName:fe26a767-49a6-4251-be25-4b62a0b0a494 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:41.551167496 +0000 UTC m=+18.216742114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcgdc" (UniqueName: "kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc") pod "network-check-target-z2lgx" (UID: "fe26a767-49a6-4251-be25-4b62a0b0a494") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:33.796305 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:33.796212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:33.796477 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.796322 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:33.796885 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:33.796729 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:33.796885 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:33.796852 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:34.795653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:34.795614 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:34.796153 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:34.795782 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:35.795000 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:35.794963 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:35.795178 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:35.795096 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:35.795451 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:35.795419 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:35.795556 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:35.795537 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:36.795281 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:36.795245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:36.795705 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:36.795375 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:37.795246 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:37.795207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:37.795467 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:37.795350 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:37.795467 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:37.795407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:37.795855 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:37.795521 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:38.795514 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:38.795486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:38.795921 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:38.795598 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:39.795868 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:39.795835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:39.796256 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:39.795843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:39.796256 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:39.795972 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:39.796256 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:39.796050 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:40.794972 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:40.794936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:40.795174 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:40.795054 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:41.415654 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:41.415612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:41.416087 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.415755 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:41.416087 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.415829 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret podName:6a93307e-c6da-4166-861e-74ad20b421b0 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:57.41580919 +0000 UTC m=+34.081383808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret") pod "global-pull-secret-syncer-jm92j" (UID: "6a93307e-c6da-4166-861e-74ad20b421b0") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:41.516810 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:41.516774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:41.516988 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.516906 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:41.516988 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.516965 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:57.516950927 +0000 UTC m=+34.182525542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:41.617289 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:41.617237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:41.617479 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.617409 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:17:41.617479 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.617430 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:17:41.617479 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.617440 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tcgdc for pod openshift-network-diagnostics/network-check-target-z2lgx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:41.617633 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.617494 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc podName:fe26a767-49a6-4251-be25-4b62a0b0a494 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:57.617478394 +0000 UTC m=+34.283053012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcgdc" (UniqueName: "kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc") pod "network-check-target-z2lgx" (UID: "fe26a767-49a6-4251-be25-4b62a0b0a494") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:41.795192 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:41.795106 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:41.795192 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:41.795127 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:41.795454 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.795235 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:41.795454 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:41.795370 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:42.795326 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:42.795241 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:42.795783 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:42.795380 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:43.796133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:43.796098 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:43.796575 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:43.796190 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:43.796575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:43.796254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:43.796575 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:43.796369 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:44.794997 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:44.794956 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:44.795141 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:44.795107 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:45.795369 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:45.795337 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:45.795986 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:45.795477 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:45.795986 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:45.795530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:45.795986 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:45.795669 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:46.795181 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:46.794853 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:46.795312 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:46.795291 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:47.054457 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.054426 2574 generic.go:356] "Generic (PLEG): container finished" podID="5fc258ae-e5cd-4f16-84bf-cd2baf654e1a" containerID="dfe746fc3bf82bb4278a0bd1ebc51cb4a54546edf0f2dec163e8a87b57acec5e" exitCode=0 Jul 24 17:17:47.055545 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.054497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mb279" event={"ID":"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a","Type":"ContainerDied","Data":"dfe746fc3bf82bb4278a0bd1ebc51cb4a54546edf0f2dec163e8a87b57acec5e"} Jul 24 17:17:47.056103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.056065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z5h8g" event={"ID":"4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31","Type":"ContainerStarted","Data":"ba9a336a39b64d2a63472d3aef936a543fe779780bbcd2382d33cde676aa1aa1"} Jul 24 17:17:47.057567 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.057532 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9t9vf" event={"ID":"6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91","Type":"ContainerStarted","Data":"2722fd2bed01ac14b79ad4986930fd0c60f50291b5acebbffb4cad8e9b676379"} Jul 24 17:17:47.059034 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.059004 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4n6l" event={"ID":"4c3a820a-6416-462c-a681-f0aa4da52e8c","Type":"ContainerStarted","Data":"b36b81330bffd5c7928d1dc2785d163f9369a009e18dcebce2622ae9754a4d74"} Jul 24 17:17:47.062237 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.062218 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" event={"ID":"0f16ccb7-8012-4838-9b38-90dabf3e05b0","Type":"ContainerStarted","Data":"8023ad1efcd16fb4e8ec1c8e5391b4cb88060fdfc86959a69a80bf97be836a5e"} Jul 24 17:17:47.062336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.062278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" event={"ID":"0f16ccb7-8012-4838-9b38-90dabf3e05b0","Type":"ContainerStarted","Data":"b4282076b0aee816852f295b73b3db979be91cc191f8b8e2f153c844a228086a"} Jul 24 17:17:47.062336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.062309 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" event={"ID":"0f16ccb7-8012-4838-9b38-90dabf3e05b0","Type":"ContainerStarted","Data":"9b018a8a419a15f21c37ed29759ce70f1b23f59d4a90809cb0108c5406d7366c"} Jul 24 17:17:47.062413 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.062342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" event={"ID":"0f16ccb7-8012-4838-9b38-90dabf3e05b0","Type":"ContainerStarted","Data":"37fda1dad6b5de9b5d1e034c7d9b87912780b831f19648abde65f47bf2b8ff97"} Jul 24 17:17:47.062413 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.062372 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" event={"ID":"0f16ccb7-8012-4838-9b38-90dabf3e05b0","Type":"ContainerStarted","Data":"b15eb98290e3e6d8cbe3982427f155c6e7a8e3ebcea34b8e2ecc3687ce6cb0d3"} Jul 24 17:17:47.062494 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.062433 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" event={"ID":"0f16ccb7-8012-4838-9b38-90dabf3e05b0","Type":"ContainerStarted","Data":"ed5912ed5d1eba573428f5ad7b7d632a515672f4914df6f7117c31eda8955c38"} Jul 24 17:17:47.063901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.063863 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxrt4" event={"ID":"04f63ec9-fb88-475f-b6f2-89ccec28afe9","Type":"ContainerStarted","Data":"d26d5c516ec9f74eb27e0ab87755d1d13a8015f48a07031fa8f7437232f2f629"} Jul 24 17:17:47.065698 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.065433 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" event={"ID":"f74c8892-223e-4771-ab9f-34954297b06d","Type":"ContainerStarted","Data":"b58429184a0345ac90bacbdbdc4a69a64d275cff5cd5dc7fa5302e8f6a7119f1"} Jul 24 17:17:47.067821 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.067800 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" event={"ID":"e4e8e0d2-7d16-49b4-86af-9556d854586d","Type":"ContainerStarted","Data":"e43eea5399d7015df6a03151ba07b2bd47b0a007bfde37788e22890cd29d1387"} Jul 24 17:17:47.077506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.077465 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-58.ec2.internal" podStartSLOduration=22.077428881 podStartE2EDuration="22.077428881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:29.030119517 +0000 UTC m=+5.695694158" watchObservedRunningTime="2026-07-24 17:17:47.077428881 +0000 UTC m=+23.743003515" Jul 24 17:17:47.096689 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.096640 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dxrt4" podStartSLOduration=3.560201703 podStartE2EDuration="23.096621202s" totalImagesPullingTime="19.536419499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:47.095847238 +0000 UTC m=+23.761421876" watchObservedRunningTime="2026-07-24 17:17:47.096621202 +0000 UTC m=+23.762195839" Jul 24 17:17:47.109759 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.109696 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c4n6l" podStartSLOduration=3.607821278 podStartE2EDuration="23.109682218s" totalImagesPullingTime="19.50186094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:47.109621324 +0000 UTC m=+23.775195976" watchObservedRunningTime="2026-07-24 17:17:47.109682218 +0000 UTC m=+23.775256853" Jul 24 17:17:47.124811 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.124763 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z5h8g" podStartSLOduration=7.591261938 podStartE2EDuration="23.124746601s" totalImagesPullingTime="15.533484663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:47.124279973 +0000 UTC m=+23.789854613" watchObservedRunningTime="2026-07-24 17:17:47.124746601 +0000 UTC m=+23.790321235" Jul 24 17:17:47.140341 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.140296 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9t9vf" podStartSLOduration=3.6681795900000003 podStartE2EDuration="23.140281148s" totalImagesPullingTime="19.472101558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:47.14027191 +0000 UTC m=+23.805846548" watchObservedRunningTime="2026-07-24 17:17:47.140281148 +0000 UTC m=+23.805855792" Jul 24 17:17:47.795136 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.795109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:47.795336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.795169 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:47.795336 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:47.795231 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:47.795451 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:47.795361 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:47.996442 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:47.996418 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 17:17:48.072806 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:48.072771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" event={"ID":"e4e8e0d2-7d16-49b4-86af-9556d854586d","Type":"ContainerStarted","Data":"382418535cca66740a2638b1261790f962996e416067e410029c7fb87c7a4f78"} Jul 24 17:17:48.074368 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:48.074333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rsvkh" event={"ID":"67675beb-50d9-4473-9216-82ec5f9b78a5","Type":"ContainerStarted","Data":"69f657e3d4d6f7598cc0c35720f7ff072afd77f583cb48e9a2b6dfeeaec8da72"} Jul 24 17:17:48.090524 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:48.090475 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rsvkh" podStartSLOduration=4.633730073 podStartE2EDuration="24.090461914s" totalImagesPullingTime="19.456731841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:48.09016922 +0000 UTC m=+24.755743893" watchObservedRunningTime="2026-07-24 17:17:48.090461914 +0000 UTC m=+24.756036550" Jul 24 17:17:48.090783 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:48.090756 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hjt6l" podStartSLOduration=4.6150970000000004 podStartE2EDuration="24.090749713s" totalImagesPullingTime="19.475652713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:47.157568265 +0000 UTC m=+23.823142906" watchObservedRunningTime="2026-07-24 17:17:48.090749713 +0000 UTC m=+24.756324348" Jul 24 17:17:48.726164 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:48.724554 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T17:17:47.996437895Z","UUID":"884ff434-3a0e-4b4e-aa60-aff595889370","Handler":null,"Name":"","Endpoint":""} Jul 24 17:17:48.726328 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:48.726250 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 17:17:48.726328 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:48.726285 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 17:17:48.794933 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:48.794903 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:48.795124 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:48.795030 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:49.082206 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:49.082174 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" event={"ID":"0f16ccb7-8012-4838-9b38-90dabf3e05b0","Type":"ContainerStarted","Data":"77c8d9b3405af162c8ef5fff8ea8aefb534c4f4704fd84744738627aca3f16dd"} Jul 24 17:17:49.795298 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:49.795260 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:49.795506 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:49.795398 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:49.795506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:49.795462 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:49.795620 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:49.795597 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:50.085838 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:50.085763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" event={"ID":"e4e8e0d2-7d16-49b4-86af-9556d854586d","Type":"ContainerStarted","Data":"51f3d55415de49ec34f0eb2c4b726d43d0462abd0a0067cb8fe2dea19dffe8fc"} Jul 24 17:17:50.795309 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:50.795273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:50.795480 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:50.795409 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:51.766663 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:51.766161 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:51.767110 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:51.766805 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:51.786900 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:51.786843 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-txbs5" podStartSLOduration=6.008853381 podStartE2EDuration="27.78682605s" totalImagesPullingTime="21.777972669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:50.111261712 +0000 UTC m=+26.776836349" watchObservedRunningTime="2026-07-24 17:17:51.78682605 +0000 UTC m=+28.452400681" Jul 24 17:17:51.795334 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:51.795312 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:51.795467 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:51.795440 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:51.795522 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:51.795480 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:51.795591 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:51.795571 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:52.090099 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:52.090075 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:52.090577 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:52.090562 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z5h8g" Jul 24 17:17:52.794860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:52.794826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:52.795514 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:52.794929 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:53.094138 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.093891 2574 generic.go:356] "Generic (PLEG): container finished" podID="5fc258ae-e5cd-4f16-84bf-cd2baf654e1a" containerID="131581a40800b0d2c0cb0dba1ac62f2ae38d69fc4bf4b71913a2ab0e03ad940f" exitCode=0 Jul 24 17:17:53.094138 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.093986 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mb279" event={"ID":"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a","Type":"ContainerDied","Data":"131581a40800b0d2c0cb0dba1ac62f2ae38d69fc4bf4b71913a2ab0e03ad940f"} Jul 24 17:17:53.097881 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.097858 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" event={"ID":"0f16ccb7-8012-4838-9b38-90dabf3e05b0","Type":"ContainerStarted","Data":"a95b9d6a7205ff0ea0e84c28ae0da46fdb982697a656ad0fabce44206f726f35"} Jul 24 17:17:53.098054 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.098033 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:53.098135 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.098079 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:53.098135 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.098125 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:53.113875 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.113855 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:53.113979 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.113968 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:17:53.142146 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.142108 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" podStartSLOduration=9.563779837 podStartE2EDuration="29.142096179s" totalImagesPullingTime="19.578316342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:17:53.140055015 +0000 UTC m=+29.805629654" watchObservedRunningTime="2026-07-24 17:17:53.142096179 +0000 UTC m=+29.807670842" Jul 24 17:17:53.796630 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.796602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:53.797080 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:53.796691 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:53.797080 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:53.796758 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:53.797080 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:53.796881 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:54.446258 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:54.446177 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6t9nt"] Jul 24 17:17:54.446421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:54.446302 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:54.446467 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:54.446438 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:54.454312 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:54.454278 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2lgx"] Jul 24 17:17:54.454441 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:54.454407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:54.454513 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:54.454494 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:54.455200 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:54.455174 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jm92j"] Jul 24 17:17:54.455296 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:54.455276 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:54.455391 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:54.455368 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:55.104780 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:55.104748 2574 generic.go:356] "Generic (PLEG): container finished" podID="5fc258ae-e5cd-4f16-84bf-cd2baf654e1a" containerID="b2dc399c5903686cd91ccf4f1b4533ede548ef3f56de6d351e797fd3d85d73ef" exitCode=0 Jul 24 17:17:55.105209 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:55.104816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mb279" event={"ID":"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a","Type":"ContainerDied","Data":"b2dc399c5903686cd91ccf4f1b4533ede548ef3f56de6d351e797fd3d85d73ef"} Jul 24 17:17:55.797652 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:55.797622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:55.797860 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:55.797761 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:55.798087 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:55.798069 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:55.798146 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:55.798133 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:55.798200 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:55.798163 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:55.798252 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:55.798203 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:56.109615 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:56.109369 2574 generic.go:356] "Generic (PLEG): container finished" podID="5fc258ae-e5cd-4f16-84bf-cd2baf654e1a" containerID="f2ad8c0ef267e1923a10526004bb34348847f56958b365adee172158baa7e883" exitCode=0 Jul 24 17:17:56.109615 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:56.109450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mb279" event={"ID":"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a","Type":"ContainerDied","Data":"f2ad8c0ef267e1923a10526004bb34348847f56958b365adee172158baa7e883"} Jul 24 17:17:57.440040 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:57.439952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:57.440401 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.440101 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:57.440401 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.440157 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret podName:6a93307e-c6da-4166-861e-74ad20b421b0 nodeName:}" failed. No retries permitted until 2026-07-24 17:18:29.440142371 +0000 UTC m=+66.105716988 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret") pod "global-pull-secret-syncer-jm92j" (UID: "6a93307e-c6da-4166-861e-74ad20b421b0") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:17:57.541190 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:57.541158 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:57.541370 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.541276 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:57.541370 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.541327 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:18:29.541310913 +0000 UTC m=+66.206885529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:17:57.642194 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:57.642163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:57.642347 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.642329 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:17:57.642388 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.642350 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:17:57.642388 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.642360 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tcgdc for pod openshift-network-diagnostics/network-check-target-z2lgx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:57.642448 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.642405 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc podName:fe26a767-49a6-4251-be25-4b62a0b0a494 nodeName:}" failed. No retries permitted until 2026-07-24 17:18:29.642391913 +0000 UTC m=+66.307966528 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tcgdc" (UniqueName: "kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc") pod "network-check-target-z2lgx" (UID: "fe26a767-49a6-4251-be25-4b62a0b0a494") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:17:57.795177 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:57.795145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:57.795364 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:57.795187 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:57.795364 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:57.795145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:57.795364 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.795258 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2lgx" podUID="fe26a767-49a6-4251-be25-4b62a0b0a494" Jul 24 17:17:57.795364 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.795331 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jm92j" podUID="6a93307e-c6da-4166-861e-74ad20b421b0" Jul 24 17:17:57.795529 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:57.795384 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:17:59.154569 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.154535 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-58.ec2.internal" event="NodeReady" Jul 24 17:17:59.154973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.154676 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 17:17:59.191945 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.191916 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fcdc7c484-ccpzp"] Jul 24 17:17:59.194391 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.194367 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.197260 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.197237 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 17:17:59.197442 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.197427 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 17:17:59.197614 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.197598 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 17:17:59.197875 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.197860 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p7fvc\"" Jul 24 17:17:59.202060 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.202031 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dbmj4"] Jul 24 17:17:59.204194 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.204175 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p6zz5"] Jul 24 17:17:59.204476 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.204454 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.206013 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.205988 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p6zz5" Jul 24 17:17:59.206534 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.206514 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 17:17:59.207092 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.207075 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 17:17:59.207179 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.207162 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 17:17:59.207313 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.207292 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2b86\"" Jul 24 17:17:59.209168 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.209150 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 17:17:59.209453 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.209337 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tlf5h\"" Jul 24 17:17:59.209677 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.209661 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 17:17:59.209765 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.209685 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 17:17:59.210260 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.210242 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fcdc7c484-ccpzp"] Jul 24 17:17:59.216185 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.216166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p6zz5"] Jul 24 17:17:59.216830 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.216811 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbmj4"] Jul 24 17:17:59.355200 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.355386 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff664f7f-5d45-4149-8c35-4d70dbc8d401-tmp-dir\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.355386 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-installation-pull-secrets\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.355386 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zvf\" (UniqueName: \"kubernetes.io/projected/ff664f7f-5d45-4149-8c35-4d70dbc8d401-kube-api-access-w2zvf\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.355386 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-bound-sa-token\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.355386 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff664f7f-5d45-4149-8c35-4d70dbc8d401-config-volume\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.355633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-certificates\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.355633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-trusted-ca\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.355633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-image-registry-private-configuration\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.355633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f394829-084a-4947-8ac4-699f1b9708fd-cert\") pod \"ingress-canary-p6zz5\" (UID: \"2f394829-084a-4947-8ac4-699f1b9708fd\") " pod="openshift-ingress-canary/ingress-canary-p6zz5" Jul 24 17:17:59.355633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dnk\" (UniqueName: \"kubernetes.io/projected/2f394829-084a-4947-8ac4-699f1b9708fd-kube-api-access-h5dnk\") pod \"ingress-canary-p6zz5\" (UID: \"2f394829-084a-4947-8ac4-699f1b9708fd\") " pod="openshift-ingress-canary/ingress-canary-p6zz5" Jul 24 17:17:59.355633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6wp\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-kube-api-access-wv6wp\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.355633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.355918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.355693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99419e6a-9ace-4182-aedb-c4f0b350a36c-ca-trust-extracted\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.456467 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff664f7f-5d45-4149-8c35-4d70dbc8d401-tmp-dir\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.456467 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-installation-pull-secrets\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.456718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zvf\" (UniqueName: \"kubernetes.io/projected/ff664f7f-5d45-4149-8c35-4d70dbc8d401-kube-api-access-w2zvf\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.456718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-bound-sa-token\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.456718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff664f7f-5d45-4149-8c35-4d70dbc8d401-config-volume\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.456718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456574 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-certificates\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.456718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456600 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-trusted-ca\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.456718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-image-registry-private-configuration\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.456718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456661 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f394829-084a-4947-8ac4-699f1b9708fd-cert\") pod \"ingress-canary-p6zz5\" (UID: \"2f394829-084a-4947-8ac4-699f1b9708fd\") " pod="openshift-ingress-canary/ingress-canary-p6zz5" Jul 24 17:17:59.456718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dnk\" (UniqueName: \"kubernetes.io/projected/2f394829-084a-4947-8ac4-699f1b9708fd-kube-api-access-h5dnk\") pod \"ingress-canary-p6zz5\" (UID: \"2f394829-084a-4947-8ac4-699f1b9708fd\") " pod="openshift-ingress-canary/ingress-canary-p6zz5" Jul 24 17:17:59.456718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456709 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6wp\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-kube-api-access-wv6wp\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.457090 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.457090 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99419e6a-9ace-4182-aedb-c4f0b350a36c-ca-trust-extracted\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.457090 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456814 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.457090 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.456843 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff664f7f-5d45-4149-8c35-4d70dbc8d401-tmp-dir\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.457090 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.456914 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:17:59.457090 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.456928 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcdc7c484-ccpzp: secret "image-registry-tls" not found Jul 24 17:17:59.457090 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.456982 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls podName:99419e6a-9ace-4182-aedb-c4f0b350a36c nodeName:}" failed. No retries permitted until 2026-07-24 17:17:59.956963193 +0000 UTC m=+36.622537830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls") pod "image-registry-6fcdc7c484-ccpzp" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c") : secret "image-registry-tls" not found Jul 24 17:17:59.457444 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.457423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99419e6a-9ace-4182-aedb-c4f0b350a36c-ca-trust-extracted\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.457508 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.457497 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:17:59.457560 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.457534 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls podName:ff664f7f-5d45-4149-8c35-4d70dbc8d401 nodeName:}" failed. No retries permitted until 2026-07-24 17:17:59.957519891 +0000 UTC m=+36.623094506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls") pod "dns-default-dbmj4" (UID: "ff664f7f-5d45-4149-8c35-4d70dbc8d401") : secret "dns-default-metrics-tls" not found Jul 24 17:17:59.457684 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.457663 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-certificates\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.457941 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.457920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff664f7f-5d45-4149-8c35-4d70dbc8d401-config-volume\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.458150 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.458127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-trusted-ca\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.461837 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.461815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-image-registry-private-configuration\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.461927 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.461909 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f394829-084a-4947-8ac4-699f1b9708fd-cert\") pod \"ingress-canary-p6zz5\" (UID: \"2f394829-084a-4947-8ac4-699f1b9708fd\") " pod="openshift-ingress-canary/ingress-canary-p6zz5" Jul 24 17:17:59.462886 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.462867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-installation-pull-secrets\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.468507 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.468488 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dnk\" (UniqueName: \"kubernetes.io/projected/2f394829-084a-4947-8ac4-699f1b9708fd-kube-api-access-h5dnk\") pod \"ingress-canary-p6zz5\" (UID: \"2f394829-084a-4947-8ac4-699f1b9708fd\") " pod="openshift-ingress-canary/ingress-canary-p6zz5" Jul 24 17:17:59.468798 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.468766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zvf\" (UniqueName: \"kubernetes.io/projected/ff664f7f-5d45-4149-8c35-4d70dbc8d401-kube-api-access-w2zvf\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.470003 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.469980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-bound-sa-token\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.470142 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.470121 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6wp\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-kube-api-access-wv6wp\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.526223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.526141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p6zz5" Jul 24 17:17:59.795276 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.795200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:17:59.795432 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.795200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:17:59.795771 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.795211 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:17:59.798565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.798543 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 17:17:59.799843 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.799821 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t6c84\"" Jul 24 17:17:59.799843 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.799821 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 17:17:59.800006 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.799901 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ntfrz\"" Jul 24 17:17:59.800006 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.799916 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 17:17:59.800113 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.800061 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 17:17:59.960839 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.960797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:17:59.961028 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:17:59.960860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:17:59.961028 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.960940 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:17:59.961028 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.960984 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:17:59.961028 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.961000 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcdc7c484-ccpzp: secret "image-registry-tls" not found Jul 24 17:17:59.961028 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.961024 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls podName:ff664f7f-5d45-4149-8c35-4d70dbc8d401 nodeName:}" failed. No retries permitted until 2026-07-24 17:18:00.961004626 +0000 UTC m=+37.626579241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls") pod "dns-default-dbmj4" (UID: "ff664f7f-5d45-4149-8c35-4d70dbc8d401") : secret "dns-default-metrics-tls" not found Jul 24 17:17:59.961274 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:17:59.961046 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls podName:99419e6a-9ace-4182-aedb-c4f0b350a36c nodeName:}" failed. No retries permitted until 2026-07-24 17:18:00.961032639 +0000 UTC m=+37.626607258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls") pod "image-registry-6fcdc7c484-ccpzp" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c") : secret "image-registry-tls" not found Jul 24 17:18:00.371721 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:00.371690 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p6zz5"] Jul 24 17:18:00.427827 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:18:00.427791 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f394829_084a_4947_8ac4_699f1b9708fd.slice/crio-46d7fffa5126665060c27c5199dd2afd0796cea056770f5d8fd49cf8a379a84c WatchSource:0}: Error finding container 46d7fffa5126665060c27c5199dd2afd0796cea056770f5d8fd49cf8a379a84c: Status 404 returned error can't find the container with id 46d7fffa5126665060c27c5199dd2afd0796cea056770f5d8fd49cf8a379a84c Jul 24 17:18:00.968272 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:00.968087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:18:00.968464 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:00.968238 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:18:00.968464 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:00.968352 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcdc7c484-ccpzp: secret "image-registry-tls" not found Jul 24 17:18:00.968464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:00.968364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:18:00.968464 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:00.968399 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls podName:99419e6a-9ace-4182-aedb-c4f0b350a36c nodeName:}" failed. No retries permitted until 2026-07-24 17:18:02.968385011 +0000 UTC m=+39.633959627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls") pod "image-registry-6fcdc7c484-ccpzp" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c") : secret "image-registry-tls" not found Jul 24 17:18:00.968639 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:00.968474 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:18:00.968639 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:00.968542 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls podName:ff664f7f-5d45-4149-8c35-4d70dbc8d401 nodeName:}" failed. No retries permitted until 2026-07-24 17:18:02.968525799 +0000 UTC m=+39.634100417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls") pod "dns-default-dbmj4" (UID: "ff664f7f-5d45-4149-8c35-4d70dbc8d401") : secret "dns-default-metrics-tls" not found Jul 24 17:18:01.123630 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:01.123593 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p6zz5" event={"ID":"2f394829-084a-4947-8ac4-699f1b9708fd","Type":"ContainerStarted","Data":"46d7fffa5126665060c27c5199dd2afd0796cea056770f5d8fd49cf8a379a84c"} Jul 24 17:18:01.126551 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:01.126526 2574 generic.go:356] "Generic (PLEG): container finished" podID="5fc258ae-e5cd-4f16-84bf-cd2baf654e1a" containerID="16f0e3a1e019c4a8d75c9e53b8275018f9105cbfa6d76a8c2c12a90c06d72284" exitCode=0 Jul 24 17:18:01.126652 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:01.126563 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mb279" event={"ID":"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a","Type":"ContainerDied","Data":"16f0e3a1e019c4a8d75c9e53b8275018f9105cbfa6d76a8c2c12a90c06d72284"} Jul 24 17:18:02.131833 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:02.131800 2574 generic.go:356] "Generic (PLEG): container finished" podID="5fc258ae-e5cd-4f16-84bf-cd2baf654e1a" containerID="1fa013d5e08a180d70193d55041e8b517ab0dd5ced762adfd52e55c6f8cffa30" exitCode=0 Jul 24 17:18:02.132278 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:02.131863 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mb279" event={"ID":"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a","Type":"ContainerDied","Data":"1fa013d5e08a180d70193d55041e8b517ab0dd5ced762adfd52e55c6f8cffa30"} Jul 24 17:18:02.986474 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:02.986441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:18:02.986641 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:02.986520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:18:02.986641 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:02.986588 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:18:02.986641 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:02.986607 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcdc7c484-ccpzp: secret "image-registry-tls" not found Jul 24 17:18:02.986641 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:02.986608 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:18:02.986862 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:02.986662 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls podName:99419e6a-9ace-4182-aedb-c4f0b350a36c nodeName:}" failed. No retries permitted until 2026-07-24 17:18:06.986646792 +0000 UTC m=+43.652221407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls") pod "image-registry-6fcdc7c484-ccpzp" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c") : secret "image-registry-tls" not found Jul 24 17:18:02.986862 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:02.986675 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls podName:ff664f7f-5d45-4149-8c35-4d70dbc8d401 nodeName:}" failed. No retries permitted until 2026-07-24 17:18:06.986669392 +0000 UTC m=+43.652244007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls") pod "dns-default-dbmj4" (UID: "ff664f7f-5d45-4149-8c35-4d70dbc8d401") : secret "dns-default-metrics-tls" not found Jul 24 17:18:03.138033 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:03.138002 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mb279" event={"ID":"5fc258ae-e5cd-4f16-84bf-cd2baf654e1a","Type":"ContainerStarted","Data":"39682fd55019738b50607197cd0f792e3f975ad8b7969fe981677d219cfaacfd"} Jul 24 17:18:03.162376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:03.162333 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mb279" podStartSLOduration=8.657283576 podStartE2EDuration="39.162317447s" totalImagesPullingTime="30.505033871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:18:03.16029441 +0000 UTC m=+39.825869047" watchObservedRunningTime="2026-07-24 17:18:03.162317447 +0000 UTC m=+39.827892104" Jul 24 17:18:04.141592 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:04.141359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p6zz5" event={"ID":"2f394829-084a-4947-8ac4-699f1b9708fd","Type":"ContainerStarted","Data":"1c8c8272442846aab668a0e5526033fab281f8a1336cec766e224b30b44b5794"} Jul 24 17:18:04.160062 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:04.160018 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p6zz5" podStartSLOduration=2.509284326 podStartE2EDuration="5.160002742s" totalImagesPullingTime="2.650718416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:18:04.159244719 +0000 UTC m=+40.824819348" watchObservedRunningTime="2026-07-24 17:18:04.160002742 +0000 UTC m=+40.825577382" Jul 24 17:18:07.016387 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:07.016341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:18:07.016387 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:07.016388 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:18:07.016887 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:07.016504 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:18:07.016887 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:07.016516 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcdc7c484-ccpzp: secret "image-registry-tls" not found Jul 24 17:18:07.016887 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:07.016514 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:18:07.016887 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:07.016568 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls podName:99419e6a-9ace-4182-aedb-c4f0b350a36c nodeName:}" failed. No retries permitted until 2026-07-24 17:18:15.016553763 +0000 UTC m=+51.682128378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls") pod "image-registry-6fcdc7c484-ccpzp" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c") : secret "image-registry-tls" not found Jul 24 17:18:07.016887 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:07.016580 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls podName:ff664f7f-5d45-4149-8c35-4d70dbc8d401 nodeName:}" failed. No retries permitted until 2026-07-24 17:18:15.016574949 +0000 UTC m=+51.682149564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls") pod "dns-default-dbmj4" (UID: "ff664f7f-5d45-4149-8c35-4d70dbc8d401") : secret "dns-default-metrics-tls" not found Jul 24 17:18:15.064768 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:15.064712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:18:15.064768 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:15.064775 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:18:15.065229 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:15.064866 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:18:15.065229 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:15.064884 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:18:15.065229 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:15.064894 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcdc7c484-ccpzp: secret "image-registry-tls" not found Jul 24 17:18:15.065229 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:15.064943 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls podName:99419e6a-9ace-4182-aedb-c4f0b350a36c nodeName:}" failed. No retries permitted until 2026-07-24 17:18:31.064925153 +0000 UTC m=+67.730499768 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls") pod "image-registry-6fcdc7c484-ccpzp" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c") : secret "image-registry-tls" not found Jul 24 17:18:15.065229 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:15.064964 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls podName:ff664f7f-5d45-4149-8c35-4d70dbc8d401 nodeName:}" failed. No retries permitted until 2026-07-24 17:18:31.064949761 +0000 UTC m=+67.730524376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls") pod "dns-default-dbmj4" (UID: "ff664f7f-5d45-4149-8c35-4d70dbc8d401") : secret "dns-default-metrics-tls" not found Jul 24 17:18:25.116191 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:25.116164 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qnk2q" Jul 24 17:18:29.459530 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.459490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:18:29.463096 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.463074 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 17:18:29.474492 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.474460 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a93307e-c6da-4166-861e-74ad20b421b0-original-pull-secret\") pod \"global-pull-secret-syncer-jm92j\" (UID: \"6a93307e-c6da-4166-861e-74ad20b421b0\") " pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:18:29.513707 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.513685 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jm92j" Jul 24 17:18:29.560197 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.560166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:18:29.563217 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.563190 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 17:18:29.570683 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:29.570661 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 17:18:29.570821 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:29.570756 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:19:33.570713294 +0000 UTC m=+130.236287919 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : secret "metrics-daemon-secret" not found Jul 24 17:18:29.634618 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.634585 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jm92j"] Jul 24 17:18:29.639277 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:18:29.639250 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a93307e_c6da_4166_861e_74ad20b421b0.slice/crio-b3e84cffcb95897fa33b457cc108b3d3b22d2eea0e1c26055d0ac2cfb7b94d07 WatchSource:0}: Error finding container b3e84cffcb95897fa33b457cc108b3d3b22d2eea0e1c26055d0ac2cfb7b94d07: Status 404 returned error can't find the container with id b3e84cffcb95897fa33b457cc108b3d3b22d2eea0e1c26055d0ac2cfb7b94d07 Jul 24 17:18:29.661125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.661103 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:18:29.664029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.664007 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 17:18:29.674619 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.674600 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 17:18:29.684962 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.684938 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgdc\" (UniqueName: \"kubernetes.io/projected/fe26a767-49a6-4251-be25-4b62a0b0a494-kube-api-access-tcgdc\") pod \"network-check-target-z2lgx\" (UID: \"fe26a767-49a6-4251-be25-4b62a0b0a494\") " pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:18:29.822384 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.822329 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t6c84\"" Jul 24 17:18:29.829853 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.829834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:18:29.946321 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:29.946292 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2lgx"] Jul 24 17:18:29.949491 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:18:29.949459 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe26a767_49a6_4251_be25_4b62a0b0a494.slice/crio-8e989780930aedd77481f23bc55d208c54f95587a6b0856cf008136b21dc296a WatchSource:0}: Error finding container 8e989780930aedd77481f23bc55d208c54f95587a6b0856cf008136b21dc296a: Status 404 returned error can't find the container with id 8e989780930aedd77481f23bc55d208c54f95587a6b0856cf008136b21dc296a Jul 24 17:18:30.209500 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:30.209459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2lgx" event={"ID":"fe26a767-49a6-4251-be25-4b62a0b0a494","Type":"ContainerStarted","Data":"8e989780930aedd77481f23bc55d208c54f95587a6b0856cf008136b21dc296a"} Jul 24 17:18:30.210687 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:30.210649 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jm92j" event={"ID":"6a93307e-c6da-4166-861e-74ad20b421b0","Type":"ContainerStarted","Data":"b3e84cffcb95897fa33b457cc108b3d3b22d2eea0e1c26055d0ac2cfb7b94d07"} Jul 24 17:18:31.071908 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:31.071855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:18:31.072314 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:31.071920 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:18:31.072314 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:31.072100 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:18:31.072314 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:31.072116 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcdc7c484-ccpzp: secret "image-registry-tls" not found Jul 24 17:18:31.072314 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:31.072122 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:18:31.072314 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:31.072181 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls podName:ff664f7f-5d45-4149-8c35-4d70dbc8d401 nodeName:}" failed. No retries permitted until 2026-07-24 17:19:03.072161439 +0000 UTC m=+99.737736054 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls") pod "dns-default-dbmj4" (UID: "ff664f7f-5d45-4149-8c35-4d70dbc8d401") : secret "dns-default-metrics-tls" not found Jul 24 17:18:31.072314 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:18:31.072207 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls podName:99419e6a-9ace-4182-aedb-c4f0b350a36c nodeName:}" failed. No retries permitted until 2026-07-24 17:19:03.07219695 +0000 UTC m=+99.737771568 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls") pod "image-registry-6fcdc7c484-ccpzp" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c") : secret "image-registry-tls" not found Jul 24 17:18:35.225164 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:35.225119 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2lgx" event={"ID":"fe26a767-49a6-4251-be25-4b62a0b0a494","Type":"ContainerStarted","Data":"b00546fd79593cde67ce41f8529bedd79689ce8aeb7fcb730040af054f57ecfe"} Jul 24 17:18:35.225687 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:35.225219 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:18:35.226470 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:35.226453 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jm92j" event={"ID":"6a93307e-c6da-4166-861e-74ad20b421b0","Type":"ContainerStarted","Data":"df9af6c7ab7a1fb9514c6f40d17f37b7fd2922a351f1d1666167681eb61a1959"} Jul 24 17:18:35.241485 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:35.241444 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z2lgx" podStartSLOduration=66.75520904 podStartE2EDuration="1m11.241430661s" totalImagesPullingTime="4.486221621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:18:35.240320482 +0000 UTC m=+71.905895120" watchObservedRunningTime="2026-07-24 17:18:35.241430661 +0000 UTC m=+71.907005327" Jul 24 17:18:35.255031 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:18:35.254992 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jm92j" podStartSLOduration=66.454849139 podStartE2EDuration="1m11.254978325s" totalImagesPullingTime="4.800129186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:18:35.254221185 +0000 UTC m=+71.919795823" watchObservedRunningTime="2026-07-24 17:18:35.254978325 +0000 UTC m=+71.920552961" Jul 24 17:19:03.092968 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:19:03.092927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:19:03.093445 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:19:03.092994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:19:03.093445 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:19:03.093077 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:19:03.093445 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:19:03.093080 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:19:03.093445 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:19:03.093098 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcdc7c484-ccpzp: secret "image-registry-tls" not found Jul 24 17:19:03.093445 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:19:03.093136 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls podName:ff664f7f-5d45-4149-8c35-4d70dbc8d401 nodeName:}" failed. No retries permitted until 2026-07-24 17:20:07.093122536 +0000 UTC m=+163.758697151 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls") pod "dns-default-dbmj4" (UID: "ff664f7f-5d45-4149-8c35-4d70dbc8d401") : secret "dns-default-metrics-tls" not found Jul 24 17:19:03.093445 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:19:03.093155 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls podName:99419e6a-9ace-4182-aedb-c4f0b350a36c nodeName:}" failed. No retries permitted until 2026-07-24 17:20:07.093141585 +0000 UTC m=+163.758716200 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls") pod "image-registry-6fcdc7c484-ccpzp" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c") : secret "image-registry-tls" not found Jul 24 17:19:06.231022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:19:06.230989 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z2lgx" Jul 24 17:19:33.597077 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:19:33.597015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:19:33.597572 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:19:33.597174 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 17:19:33.597572 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:19:33.597252 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:35.597235983 +0000 UTC m=+252.262810598 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : secret "metrics-daemon-secret" not found Jul 24 17:20:02.207127 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:20:02.207076 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" podUID="99419e6a-9ace-4182-aedb-c4f0b350a36c" Jul 24 17:20:02.219335 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:20:02.219309 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dbmj4" podUID="ff664f7f-5d45-4149-8c35-4d70dbc8d401" Jul 24 17:20:02.432582 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:20:02.432557 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbmj4" Jul 24 17:20:02.432705 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:20:02.432561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:20:02.806220 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:20:02.806170 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:20:07.115290 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:20:07.115241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:20:07.115703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:20:07.115346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:20:07.115703 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:20:07.115398 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:20:07.115847 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:20:07.115798 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcdc7c484-ccpzp: secret "image-registry-tls" not found Jul 24 17:20:07.115847 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:20:07.115825 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:20:07.115940 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:20:07.115896 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls podName:99419e6a-9ace-4182-aedb-c4f0b350a36c nodeName:}" failed. No retries permitted until 2026-07-24 17:22:09.115862572 +0000 UTC m=+285.781437202 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls") pod "image-registry-6fcdc7c484-ccpzp" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c") : secret "image-registry-tls" not found Jul 24 17:20:07.115940 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:20:07.115918 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls podName:ff664f7f-5d45-4149-8c35-4d70dbc8d401 nodeName:}" failed. No retries permitted until 2026-07-24 17:22:09.115908126 +0000 UTC m=+285.781482741 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls") pod "dns-default-dbmj4" (UID: "ff664f7f-5d45-4149-8c35-4d70dbc8d401") : secret "dns-default-metrics-tls" not found Jul 24 17:20:17.795395 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:20:17.795306 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:21:35.603422 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:35.603365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:21:35.603948 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:35.603533 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 17:21:35.603948 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:35.603602 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs podName:bd39d23d-0f58-49d6-8d42-39caa7c31c41 nodeName:}" failed. No retries permitted until 2026-07-24 17:23:37.603585159 +0000 UTC m=+374.269159774 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs") pod "network-metrics-daemon-6t9nt" (UID: "bd39d23d-0f58-49d6-8d42-39caa7c31c41") : secret "metrics-daemon-secret" not found Jul 24 17:21:38.332764 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.332712 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5"] Jul 24 17:21:38.334567 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.334546 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.337661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.337637 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4lm9j\"" Jul 24 17:21:38.337957 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.337940 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh"] Jul 24 17:21:38.338320 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.338298 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 17:21:38.338396 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.338331 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:21:38.338396 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.338361 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 17:21:38.338572 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.338306 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 17:21:38.339563 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.339545 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh" Jul 24 17:21:38.343104 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.343085 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m"] Jul 24 17:21:38.344739 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.344708 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-64lhm"] Jul 24 17:21:38.344828 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.344793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.346336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.346318 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-67844487c7-fg86z"] Jul 24 17:21:38.346432 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.346420 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.347898 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.347878 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.347993 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.347921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:21:38.347993 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.347945 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 17:21:38.349429 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.349409 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 17:21:38.349429 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.349426 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:21:38.349572 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.349440 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8ztpb\"" Jul 24 17:21:38.349572 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.349494 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 17:21:38.349674 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.349587 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xdr7l\"" Jul 24 17:21:38.349674 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.349592 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 17:21:38.349674 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.349652 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5"] Jul 24 17:21:38.350634 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.350602 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:21:38.350634 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.350604 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 17:21:38.350864 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.350606 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 17:21:38.350864 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.350776 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 17:21:38.350864 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.350790 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5zg59\"" Jul 24 17:21:38.352012 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.351995 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 17:21:38.352102 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.352023 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 17:21:38.360020 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.359998 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 17:21:38.360898 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.360879 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kvrch\"" Jul 24 17:21:38.361795 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.361774 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 17:21:38.361906 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.361841 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 17:21:38.362018 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.362000 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 17:21:38.362648 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.362627 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh"] Jul 24 17:21:38.363431 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.363411 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-64lhm"] Jul 24 17:21:38.364327 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.364304 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m"] Jul 24 17:21:38.367198 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.367177 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 17:21:38.367827 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.367811 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67844487c7-fg86z"] Jul 24 17:21:38.421334 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.421306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6fd8a29-1f40-40e1-a279-21e27465afc4-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5\" (UID: \"f6fd8a29-1f40-40e1-a279-21e27465afc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.421502 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.421340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6fd8a29-1f40-40e1-a279-21e27465afc4-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5\" (UID: \"f6fd8a29-1f40-40e1-a279-21e27465afc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.421502 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.421366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjch\" (UniqueName: \"kubernetes.io/projected/f6fd8a29-1f40-40e1-a279-21e27465afc4-kube-api-access-lrjch\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5\" (UID: \"f6fd8a29-1f40-40e1-a279-21e27465afc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.430688 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.430659 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm"] Jul 24 17:21:38.432746 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.432716 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:38.436337 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.436310 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 17:21:38.436337 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.436324 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bq76x\"" Jul 24 17:21:38.436514 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.436347 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 17:21:38.436576 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.436516 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 17:21:38.438219 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.438197 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 17:21:38.438974 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.438956 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-hzg25"] Jul 24 17:21:38.440966 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.440949 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.453224 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.453202 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm"] Jul 24 17:21:38.453583 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.453567 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 17:21:38.457346 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.457330 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 17:21:38.457659 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.457646 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 17:21:38.458501 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.458488 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 17:21:38.463289 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.463272 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-hzg25"] Jul 24 17:21:38.464047 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.464030 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6tfzh\"" Jul 24 17:21:38.485045 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.485019 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 17:21:38.522571 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glvnm\" (UniqueName: \"kubernetes.io/projected/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-kube-api-access-glvnm\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.522571 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.522775 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522588 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p5cr\" (UniqueName: \"kubernetes.io/projected/e864556f-717c-46eb-81a3-eba010817bae-kube-api-access-9p5cr\") pod \"volume-data-source-validator-549c6d54cf-t67nh\" (UID: \"e864556f-717c-46eb-81a3-eba010817bae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh" Jul 24 17:21:38.522775 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522610 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6fd8a29-1f40-40e1-a279-21e27465afc4-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5\" (UID: \"f6fd8a29-1f40-40e1-a279-21e27465afc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.522775 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3a496af-b330-41c0-a0e8-acbb1d153492-config\") pod \"service-ca-operator-5d6c75f84f-9k69m\" (UID: \"a3a496af-b330-41c0-a0e8-acbb1d153492\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.522775 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6dp\" (UniqueName: \"kubernetes.io/projected/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-kube-api-access-fz6dp\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.522775 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55sd4\" (UniqueName: \"kubernetes.io/projected/a3a496af-b330-41c0-a0e8-acbb1d153492-kube-api-access-55sd4\") pod \"service-ca-operator-5d6c75f84f-9k69m\" (UID: \"a3a496af-b330-41c0-a0e8-acbb1d153492\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.522775 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-default-certificate\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.522958 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmv86\" (UniqueName: \"kubernetes.io/projected/81f9cda0-d765-4821-b23a-62d5a9f17b12-kube-api-access-cmv86\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:38.522958 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.522958 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-serving-cert\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.522958 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a496af-b330-41c0-a0e8-acbb1d153492-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9k69m\" (UID: \"a3a496af-b330-41c0-a0e8-acbb1d153492\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.522958 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-config\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.522958 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/81f9cda0-d765-4821-b23a-62d5a9f17b12-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:38.523132 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6fd8a29-1f40-40e1-a279-21e27465afc4-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5\" (UID: \"f6fd8a29-1f40-40e1-a279-21e27465afc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.523132 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.522993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-stats-auth\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.523132 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.523014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-trusted-ca\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.523132 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.523039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjch\" (UniqueName: \"kubernetes.io/projected/f6fd8a29-1f40-40e1-a279-21e27465afc4-kube-api-access-lrjch\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5\" (UID: \"f6fd8a29-1f40-40e1-a279-21e27465afc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.523132 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.523098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:38.523276 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.523166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6fd8a29-1f40-40e1-a279-21e27465afc4-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5\" (UID: \"f6fd8a29-1f40-40e1-a279-21e27465afc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.525543 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.525519 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6fd8a29-1f40-40e1-a279-21e27465afc4-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5\" (UID: \"f6fd8a29-1f40-40e1-a279-21e27465afc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.532172 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.532151 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjch\" (UniqueName: \"kubernetes.io/projected/f6fd8a29-1f40-40e1-a279-21e27465afc4-kube-api-access-lrjch\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5\" (UID: \"f6fd8a29-1f40-40e1-a279-21e27465afc4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.623713 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3a496af-b330-41c0-a0e8-acbb1d153492-config\") pod \"service-ca-operator-5d6c75f84f-9k69m\" (UID: \"a3a496af-b330-41c0-a0e8-acbb1d153492\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.623713 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6dp\" (UniqueName: \"kubernetes.io/projected/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-kube-api-access-fz6dp\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.623713 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55sd4\" (UniqueName: \"kubernetes.io/projected/a3a496af-b330-41c0-a0e8-acbb1d153492-kube-api-access-55sd4\") pod \"service-ca-operator-5d6c75f84f-9k69m\" (UID: \"a3a496af-b330-41c0-a0e8-acbb1d153492\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.623983 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-default-certificate\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.623983 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmv86\" (UniqueName: \"kubernetes.io/projected/81f9cda0-d765-4821-b23a-62d5a9f17b12-kube-api-access-cmv86\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:38.623983 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.623983 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-serving-cert\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.623983 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a496af-b330-41c0-a0e8-acbb1d153492-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9k69m\" (UID: \"a3a496af-b330-41c0-a0e8-acbb1d153492\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.623983 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-config\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.623983 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:38.623882 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:21:38.623983 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:38.623947 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:39.123927449 +0000 UTC m=+255.789502064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : secret "router-metrics-certs-default" not found Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.623888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10aa3400-8f79-4e7f-97bb-d6ca24d01430-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624083 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/81f9cda0-d765-4821-b23a-62d5a9f17b12-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-stats-auth\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-trusted-ca\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10aa3400-8f79-4e7f-97bb-d6ca24d01430-snapshots\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10aa3400-8f79-4e7f-97bb-d6ca24d01430-serving-cert\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10aa3400-8f79-4e7f-97bb-d6ca24d01430-service-ca-bundle\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10aa3400-8f79-4e7f-97bb-d6ca24d01430-tmp\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3a496af-b330-41c0-a0e8-acbb1d153492-config\") pod \"service-ca-operator-5d6c75f84f-9k69m\" (UID: \"a3a496af-b330-41c0-a0e8-acbb1d153492\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.624367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glvnm\" (UniqueName: \"kubernetes.io/projected/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-kube-api-access-glvnm\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.625076 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jsh\" (UniqueName: \"kubernetes.io/projected/10aa3400-8f79-4e7f-97bb-d6ca24d01430-kube-api-access-85jsh\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.625076 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624615 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-config\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.625076 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/81f9cda0-d765-4821-b23a-62d5a9f17b12-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:38.625076 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.625076 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.624944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p5cr\" (UniqueName: \"kubernetes.io/projected/e864556f-717c-46eb-81a3-eba010817bae-kube-api-access-9p5cr\") pod \"volume-data-source-validator-549c6d54cf-t67nh\" (UID: \"e864556f-717c-46eb-81a3-eba010817bae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh" Jul 24 17:21:38.625076 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:38.624997 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:38.625076 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:38.625000 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:39.124984572 +0000 UTC m=+255.790559205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : configmap references non-existent config key: service-ca.crt Jul 24 17:21:38.625076 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:38.625040 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls podName:81f9cda0-d765-4821-b23a-62d5a9f17b12 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:39.12503018 +0000 UTC m=+255.790604804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ngcgm" (UID: "81f9cda0-d765-4821-b23a-62d5a9f17b12") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:38.625573 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.625549 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-trusted-ca\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.626605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.626576 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-serving-cert\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.626787 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.626768 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a496af-b330-41c0-a0e8-acbb1d153492-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9k69m\" (UID: \"a3a496af-b330-41c0-a0e8-acbb1d153492\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.626953 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.626935 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-default-certificate\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.626995 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.626965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-stats-auth\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.635272 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.635238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6dp\" (UniqueName: \"kubernetes.io/projected/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-kube-api-access-fz6dp\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:38.635676 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.635653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmv86\" (UniqueName: \"kubernetes.io/projected/81f9cda0-d765-4821-b23a-62d5a9f17b12-kube-api-access-cmv86\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:38.635765 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.635653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glvnm\" (UniqueName: \"kubernetes.io/projected/411738aa-268d-4f8d-b8bd-1f8cfc0ebff4-kube-api-access-glvnm\") pod \"console-operator-584dcb6b54-64lhm\" (UID: \"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4\") " pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.635817 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.635784 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55sd4\" (UniqueName: \"kubernetes.io/projected/a3a496af-b330-41c0-a0e8-acbb1d153492-kube-api-access-55sd4\") pod \"service-ca-operator-5d6c75f84f-9k69m\" (UID: \"a3a496af-b330-41c0-a0e8-acbb1d153492\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.636281 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.636265 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p5cr\" (UniqueName: \"kubernetes.io/projected/e864556f-717c-46eb-81a3-eba010817bae-kube-api-access-9p5cr\") pod \"volume-data-source-validator-549c6d54cf-t67nh\" (UID: \"e864556f-717c-46eb-81a3-eba010817bae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh" Jul 24 17:21:38.643488 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.643471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" Jul 24 17:21:38.651589 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.651566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh" Jul 24 17:21:38.658264 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.658240 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" Jul 24 17:21:38.667065 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.667046 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.726195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10aa3400-8f79-4e7f-97bb-d6ca24d01430-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.726257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10aa3400-8f79-4e7f-97bb-d6ca24d01430-snapshots\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.726292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10aa3400-8f79-4e7f-97bb-d6ca24d01430-serving-cert\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.726323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10aa3400-8f79-4e7f-97bb-d6ca24d01430-service-ca-bundle\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.726390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10aa3400-8f79-4e7f-97bb-d6ca24d01430-tmp\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.726462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jsh\" (UniqueName: \"kubernetes.io/projected/10aa3400-8f79-4e7f-97bb-d6ca24d01430-kube-api-access-85jsh\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.727870 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10aa3400-8f79-4e7f-97bb-d6ca24d01430-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.728414 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/10aa3400-8f79-4e7f-97bb-d6ca24d01430-snapshots\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.728754 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10aa3400-8f79-4e7f-97bb-d6ca24d01430-tmp\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.729173 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.728878 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10aa3400-8f79-4e7f-97bb-d6ca24d01430-service-ca-bundle\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.732638 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.732443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10aa3400-8f79-4e7f-97bb-d6ca24d01430-serving-cert\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.736345 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.736325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jsh\" (UniqueName: \"kubernetes.io/projected/10aa3400-8f79-4e7f-97bb-d6ca24d01430-kube-api-access-85jsh\") pod \"insights-operator-5885fdb759-hzg25\" (UID: \"10aa3400-8f79-4e7f-97bb-d6ca24d01430\") " pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.748952 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.748925 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-hzg25" Jul 24 17:21:38.798686 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.798609 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5"] Jul 24 17:21:38.830015 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.829983 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh"] Jul 24 17:21:38.839139 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.839117 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m"] Jul 24 17:21:38.852696 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:21:38.852669 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode864556f_717c_46eb_81a3_eba010817bae.slice/crio-9dd62d8df7f995488479bc33ea0836dd90b2a710a8a19f7b6d889789d2dadbed WatchSource:0}: Error finding container 9dd62d8df7f995488479bc33ea0836dd90b2a710a8a19f7b6d889789d2dadbed: Status 404 returned error can't find the container with id 9dd62d8df7f995488479bc33ea0836dd90b2a710a8a19f7b6d889789d2dadbed Jul 24 17:21:38.853414 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:21:38.853391 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a496af_b330_41c0_a0e8_acbb1d153492.slice/crio-ae5cc61ae1f7ba12a3d5d01906e2fff957a3e6f166bff1b904608b76b37caeeb WatchSource:0}: Error finding container ae5cc61ae1f7ba12a3d5d01906e2fff957a3e6f166bff1b904608b76b37caeeb: Status 404 returned error can't find the container with id ae5cc61ae1f7ba12a3d5d01906e2fff957a3e6f166bff1b904608b76b37caeeb Jul 24 17:21:38.862665 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.862629 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-64lhm"] Jul 24 17:21:38.865662 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:21:38.865643 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411738aa_268d_4f8d_b8bd_1f8cfc0ebff4.slice/crio-d4dd50be318478adff01dce8b28d3db1df93fcdd314c1070497e3e725acf1ee7 WatchSource:0}: Error finding container d4dd50be318478adff01dce8b28d3db1df93fcdd314c1070497e3e725acf1ee7: Status 404 returned error can't find the container with id d4dd50be318478adff01dce8b28d3db1df93fcdd314c1070497e3e725acf1ee7 Jul 24 17:21:38.903427 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:38.903341 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-hzg25"] Jul 24 17:21:38.907245 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:21:38.907221 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10aa3400_8f79_4e7f_97bb_d6ca24d01430.slice/crio-b2fcbe1183f2c3e7502f042f96970ac593c1ce87985751b0f1c41e9f54da3a08 WatchSource:0}: Error finding container b2fcbe1183f2c3e7502f042f96970ac593c1ce87985751b0f1c41e9f54da3a08: Status 404 returned error can't find the container with id b2fcbe1183f2c3e7502f042f96970ac593c1ce87985751b0f1c41e9f54da3a08 Jul 24 17:21:39.129756 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:39.129708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:39.129943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:39.129780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:39.129943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:39.129835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:39.129943 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:39.129877 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:39.129943 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:39.129941 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:40.129929202 +0000 UTC m=+256.795503817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : configmap references non-existent config key: service-ca.crt Jul 24 17:21:39.130098 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:39.129965 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls podName:81f9cda0-d765-4821-b23a-62d5a9f17b12 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:40.129951904 +0000 UTC m=+256.795526522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ngcgm" (UID: "81f9cda0-d765-4821-b23a-62d5a9f17b12") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:39.130098 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:39.129877 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:21:39.130098 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:39.129993 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:40.129984549 +0000 UTC m=+256.795559168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : secret "router-metrics-certs-default" not found Jul 24 17:21:39.659375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:39.659287 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-hzg25" event={"ID":"10aa3400-8f79-4e7f-97bb-d6ca24d01430","Type":"ContainerStarted","Data":"b2fcbe1183f2c3e7502f042f96970ac593c1ce87985751b0f1c41e9f54da3a08"} Jul 24 17:21:39.661945 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:39.661349 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" event={"ID":"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4","Type":"ContainerStarted","Data":"d4dd50be318478adff01dce8b28d3db1df93fcdd314c1070497e3e725acf1ee7"} Jul 24 17:21:39.663234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:39.663175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" event={"ID":"a3a496af-b330-41c0-a0e8-acbb1d153492","Type":"ContainerStarted","Data":"ae5cc61ae1f7ba12a3d5d01906e2fff957a3e6f166bff1b904608b76b37caeeb"} Jul 24 17:21:39.664691 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:39.664630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh" event={"ID":"e864556f-717c-46eb-81a3-eba010817bae","Type":"ContainerStarted","Data":"9dd62d8df7f995488479bc33ea0836dd90b2a710a8a19f7b6d889789d2dadbed"} Jul 24 17:21:39.666304 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:39.666277 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" event={"ID":"f6fd8a29-1f40-40e1-a279-21e27465afc4","Type":"ContainerStarted","Data":"75608e79cb2bbf05e34a7ba5e47b1e62205b4db68d1066a75843e738cf3675b1"} Jul 24 17:21:40.139041 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:40.139000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:40.139240 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:40.139095 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:40.139240 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:40.139145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:40.139352 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:40.139290 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:21:40.139352 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:40.139349 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:42.139331305 +0000 UTC m=+258.804905938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : secret "router-metrics-certs-default" not found Jul 24 17:21:40.139573 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:40.139463 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:40.139573 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:40.139532 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls podName:81f9cda0-d765-4821-b23a-62d5a9f17b12 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:42.139515347 +0000 UTC m=+258.805089964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ngcgm" (UID: "81f9cda0-d765-4821-b23a-62d5a9f17b12") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:40.139573 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:40.139550 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:42.139538404 +0000 UTC m=+258.805113038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : configmap references non-existent config key: service-ca.crt Jul 24 17:21:40.671432 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:40.671382 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh" event={"ID":"e864556f-717c-46eb-81a3-eba010817bae","Type":"ContainerStarted","Data":"287c376f5704f901026757d24ca820066a4e016cfa755bd4b47ef1fde4296290"} Jul 24 17:21:40.689608 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:40.689555 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-t67nh" podStartSLOduration=1.254625546 podStartE2EDuration="2.689532536s" totalImagesPullingTime="1.43490699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:40.688885893 +0000 UTC m=+257.354460532" watchObservedRunningTime="2026-07-24 17:21:40.689532536 +0000 UTC m=+257.355107173" Jul 24 17:21:42.156404 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.156365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:42.156971 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.156447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:42.156971 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.156520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:42.156971 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:42.156537 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:21:42.156971 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:42.156592 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:42.156971 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:42.156605 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:46.156588982 +0000 UTC m=+262.822163598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : secret "router-metrics-certs-default" not found Jul 24 17:21:42.156971 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:42.156654 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:46.156637366 +0000 UTC m=+262.822212006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : configmap references non-existent config key: service-ca.crt Jul 24 17:21:42.156971 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:42.156675 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls podName:81f9cda0-d765-4821-b23a-62d5a9f17b12 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:46.156664623 +0000 UTC m=+262.822239238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ngcgm" (UID: "81f9cda0-d765-4821-b23a-62d5a9f17b12") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:42.679509 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.679479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-hzg25" event={"ID":"10aa3400-8f79-4e7f-97bb-d6ca24d01430","Type":"ContainerStarted","Data":"69c96f092af258c95fde8e0585707a6dec251bdbf068ca2348a3238d99620a74"} Jul 24 17:21:42.681693 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.681670 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/0.log" Jul 24 17:21:42.681833 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.681711 2574 generic.go:356] "Generic (PLEG): container finished" podID="411738aa-268d-4f8d-b8bd-1f8cfc0ebff4" containerID="f6d1ed05794c7561364bbd5b0641eef5cb255075fe85c155700136b8a6686296" exitCode=255 Jul 24 17:21:42.681833 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.681769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" event={"ID":"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4","Type":"ContainerDied","Data":"f6d1ed05794c7561364bbd5b0641eef5cb255075fe85c155700136b8a6686296"} Jul 24 17:21:42.682041 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.682003 2574 scope.go:117] "RemoveContainer" containerID="f6d1ed05794c7561364bbd5b0641eef5cb255075fe85c155700136b8a6686296" Jul 24 17:21:42.683520 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.683497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" event={"ID":"a3a496af-b330-41c0-a0e8-acbb1d153492","Type":"ContainerStarted","Data":"f0c98e156f9edb19c84f7542adfabb0855860793c9103612b839445e321c7231"} Jul 24 17:21:42.685949 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.685922 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" event={"ID":"f6fd8a29-1f40-40e1-a279-21e27465afc4","Type":"ContainerStarted","Data":"e414d271af757f462f71437fc6ee64c4fe49a19e39689171f674ae5f47df9f6c"} Jul 24 17:21:42.722909 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.722863 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" podStartSLOduration=1.064540273 podStartE2EDuration="4.722848763s" totalImagesPullingTime="3.65830849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:42.721777013 +0000 UTC m=+259.387351652" watchObservedRunningTime="2026-07-24 17:21:42.722848763 +0000 UTC m=+259.388423408" Jul 24 17:21:42.723051 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.722933 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-hzg25" podStartSLOduration=1.161992085 podStartE2EDuration="4.722927358s" totalImagesPullingTime="3.560935273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:42.703483916 +0000 UTC m=+259.369058557" watchObservedRunningTime="2026-07-24 17:21:42.722927358 +0000 UTC m=+259.388501995" Jul 24 17:21:42.738618 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:42.738567 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" podStartSLOduration=1.122751154 podStartE2EDuration="4.738548105s" totalImagesPullingTime="3.615796951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:42.737765931 +0000 UTC m=+259.403340569" watchObservedRunningTime="2026-07-24 17:21:42.738548105 +0000 UTC m=+259.404122743" Jul 24 17:21:43.691664 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:43.691636 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/1.log" Jul 24 17:21:43.692094 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:43.692060 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/0.log" Jul 24 17:21:43.692094 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:43.692090 2574 generic.go:356] "Generic (PLEG): container finished" podID="411738aa-268d-4f8d-b8bd-1f8cfc0ebff4" containerID="f352a5cb988befad46612cbab3016037c0f6563f6448ed752e82aec09f85dc8e" exitCode=255 Jul 24 17:21:43.692217 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:43.692193 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" event={"ID":"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4","Type":"ContainerDied","Data":"f352a5cb988befad46612cbab3016037c0f6563f6448ed752e82aec09f85dc8e"} Jul 24 17:21:43.692271 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:43.692256 2574 scope.go:117] "RemoveContainer" containerID="f6d1ed05794c7561364bbd5b0641eef5cb255075fe85c155700136b8a6686296" Jul 24 17:21:43.692712 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:43.692690 2574 scope.go:117] "RemoveContainer" containerID="f352a5cb988befad46612cbab3016037c0f6563f6448ed752e82aec09f85dc8e" Jul 24 17:21:43.692884 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:43.692864 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-64lhm_openshift-console-operator(411738aa-268d-4f8d-b8bd-1f8cfc0ebff4)\"" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" podUID="411738aa-268d-4f8d-b8bd-1f8cfc0ebff4" Jul 24 17:21:44.210035 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.209952 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m5png"] Jul 24 17:21:44.212606 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.212586 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.215605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.215579 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bstvq\"" Jul 24 17:21:44.215605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.215579 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 17:21:44.216910 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.216891 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 17:21:44.227886 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.227863 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m5png"] Jul 24 17:21:44.275659 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.275629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-crio-socket\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.275839 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.275783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.275839 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.275828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lzs\" (UniqueName: \"kubernetes.io/projected/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-api-access-g4lzs\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.275910 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.275851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-data-volume\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.275910 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.275902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.376655 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.376628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.376872 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.376661 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lzs\" (UniqueName: \"kubernetes.io/projected/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-api-access-g4lzs\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.376872 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.376684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-data-volume\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.376872 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.376752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.376872 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.376784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-crio-socket\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.376872 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:44.376795 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:21:44.376872 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:44.376863 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:44.876842968 +0000 UTC m=+261.542417589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : secret "insights-runtime-extractor-tls" not found Jul 24 17:21:44.377134 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:44.376890 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:21:44.377134 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.376899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-crio-socket\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.377134 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:44.376954 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:44.87693582 +0000 UTC m=+261.542510450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : configmap "kube-rbac-proxy" not found Jul 24 17:21:44.377134 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.377091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-data-volume\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.386027 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.386001 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lzs\" (UniqueName: \"kubernetes.io/projected/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-api-access-g4lzs\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.696359 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.696278 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/1.log" Jul 24 17:21:44.696820 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.696595 2574 scope.go:117] "RemoveContainer" containerID="f352a5cb988befad46612cbab3016037c0f6563f6448ed752e82aec09f85dc8e" Jul 24 17:21:44.696820 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:44.696745 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-64lhm_openshift-console-operator(411738aa-268d-4f8d-b8bd-1f8cfc0ebff4)\"" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" podUID="411738aa-268d-4f8d-b8bd-1f8cfc0ebff4" Jul 24 17:21:44.881505 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.881460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.881703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:44.881565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:44.881703 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:44.881607 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:21:44.881703 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:44.881681 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:45.881660221 +0000 UTC m=+262.547234855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : secret "insights-runtime-extractor-tls" not found Jul 24 17:21:44.881703 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:44.881691 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:21:44.881935 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:44.881766 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:45.881749853 +0000 UTC m=+262.547324486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : configmap "kube-rbac-proxy" not found Jul 24 17:21:45.887589 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:45.887547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:45.888001 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:45.887611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:45.888001 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:45.887695 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:21:45.888001 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:45.887716 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:21:45.888001 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:45.887781 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:47.887766654 +0000 UTC m=+264.553341270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : configmap "kube-rbac-proxy" not found Jul 24 17:21:45.888001 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:45.887798 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:47.887789074 +0000 UTC m=+264.553363689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : secret "insights-runtime-extractor-tls" not found Jul 24 17:21:46.189006 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.188901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:46.189006 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.188989 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:46.189186 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.189044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:46.189186 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:46.189073 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:54.189049944 +0000 UTC m=+270.854624559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : configmap references non-existent config key: service-ca.crt Jul 24 17:21:46.189186 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:46.189131 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:46.189186 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:46.189131 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:21:46.189186 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:46.189183 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:54.189167914 +0000 UTC m=+270.854742531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : secret "router-metrics-certs-default" not found Jul 24 17:21:46.189343 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:46.189198 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls podName:81f9cda0-d765-4821-b23a-62d5a9f17b12 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:54.189191874 +0000 UTC m=+270.854766489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ngcgm" (UID: "81f9cda0-d765-4821-b23a-62d5a9f17b12") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:46.800309 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.800285 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9t9vf_6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91/dns-node-resolver/0.log" Jul 24 17:21:46.808739 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.808694 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-kg4ch"] Jul 24 17:21:46.810448 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.810433 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:46.813355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.813336 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 17:21:46.813478 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.813336 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 17:21:46.815034 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.815013 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 17:21:46.815133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.815071 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5wr4n\"" Jul 24 17:21:46.815133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.815075 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 17:21:46.819899 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.819880 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-kg4ch"] Jul 24 17:21:46.894705 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.894679 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a2d04c48-61de-4063-94c3-4132ea7e153a-signing-cabundle\") pod \"service-ca-658d7b7c78-kg4ch\" (UID: \"a2d04c48-61de-4063-94c3-4132ea7e153a\") " pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:46.895086 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.894793 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4p9\" (UniqueName: \"kubernetes.io/projected/a2d04c48-61de-4063-94c3-4132ea7e153a-kube-api-access-cl4p9\") pod \"service-ca-658d7b7c78-kg4ch\" (UID: \"a2d04c48-61de-4063-94c3-4132ea7e153a\") " pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:46.895086 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.894821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a2d04c48-61de-4063-94c3-4132ea7e153a-signing-key\") pod \"service-ca-658d7b7c78-kg4ch\" (UID: \"a2d04c48-61de-4063-94c3-4132ea7e153a\") " pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:46.995214 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.995178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4p9\" (UniqueName: \"kubernetes.io/projected/a2d04c48-61de-4063-94c3-4132ea7e153a-kube-api-access-cl4p9\") pod \"service-ca-658d7b7c78-kg4ch\" (UID: \"a2d04c48-61de-4063-94c3-4132ea7e153a\") " pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:46.995375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.995229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a2d04c48-61de-4063-94c3-4132ea7e153a-signing-key\") pod \"service-ca-658d7b7c78-kg4ch\" (UID: \"a2d04c48-61de-4063-94c3-4132ea7e153a\") " pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:46.995468 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.995445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a2d04c48-61de-4063-94c3-4132ea7e153a-signing-cabundle\") pod \"service-ca-658d7b7c78-kg4ch\" (UID: \"a2d04c48-61de-4063-94c3-4132ea7e153a\") " pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:46.996051 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.996026 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a2d04c48-61de-4063-94c3-4132ea7e153a-signing-cabundle\") pod \"service-ca-658d7b7c78-kg4ch\" (UID: \"a2d04c48-61de-4063-94c3-4132ea7e153a\") " pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:46.997605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:46.997579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a2d04c48-61de-4063-94c3-4132ea7e153a-signing-key\") pod \"service-ca-658d7b7c78-kg4ch\" (UID: \"a2d04c48-61de-4063-94c3-4132ea7e153a\") " pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:47.003251 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:47.003227 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4p9\" (UniqueName: \"kubernetes.io/projected/a2d04c48-61de-4063-94c3-4132ea7e153a-kube-api-access-cl4p9\") pod \"service-ca-658d7b7c78-kg4ch\" (UID: \"a2d04c48-61de-4063-94c3-4132ea7e153a\") " pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:47.119708 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:47.119628 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" Jul 24 17:21:47.258744 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:47.258706 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-kg4ch"] Jul 24 17:21:47.260143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:21:47.260114 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d04c48_61de_4063_94c3_4132ea7e153a.slice/crio-3832299c4ecf6001110ff3230cf53909c4fed8f2e87c3271c07af7fc77fdd48a WatchSource:0}: Error finding container 3832299c4ecf6001110ff3230cf53909c4fed8f2e87c3271c07af7fc77fdd48a: Status 404 returned error can't find the container with id 3832299c4ecf6001110ff3230cf53909c4fed8f2e87c3271c07af7fc77fdd48a Jul 24 17:21:47.708124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:47.708082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" event={"ID":"a2d04c48-61de-4063-94c3-4132ea7e153a","Type":"ContainerStarted","Data":"4e25257b154154af2bb72279b96224af33bb2bf3cefc57bc510d5f644f7d24a5"} Jul 24 17:21:47.708421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:47.708182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" event={"ID":"a2d04c48-61de-4063-94c3-4132ea7e153a","Type":"ContainerStarted","Data":"3832299c4ecf6001110ff3230cf53909c4fed8f2e87c3271c07af7fc77fdd48a"} Jul 24 17:21:47.729333 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:47.729285 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-kg4ch" podStartSLOduration=1.7292698720000002 podStartE2EDuration="1.729269872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:21:47.727080862 +0000 UTC m=+264.392655500" watchObservedRunningTime="2026-07-24 17:21:47.729269872 +0000 UTC m=+264.394844509" Jul 24 17:21:47.798026 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:47.797999 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4n6l_4c3a820a-6416-462c-a681-f0aa4da52e8c/node-ca/0.log" Jul 24 17:21:47.902573 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:47.902537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:47.903003 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:47.902609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:47.903003 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:47.902683 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:21:47.903003 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:47.902683 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:21:47.903003 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:47.902768 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:51.902750958 +0000 UTC m=+268.568325574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : configmap "kube-rbac-proxy" not found Jul 24 17:21:47.903003 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:47.902783 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:51.902776633 +0000 UTC m=+268.568351248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : secret "insights-runtime-extractor-tls" not found Jul 24 17:21:48.667515 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:48.667411 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:48.667515 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:48.667484 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:21:48.667931 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:48.667914 2574 scope.go:117] "RemoveContainer" containerID="f352a5cb988befad46612cbab3016037c0f6563f6448ed752e82aec09f85dc8e" Jul 24 17:21:48.668115 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:48.668092 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-64lhm_openshift-console-operator(411738aa-268d-4f8d-b8bd-1f8cfc0ebff4)\"" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" podUID="411738aa-268d-4f8d-b8bd-1f8cfc0ebff4" Jul 24 17:21:48.798887 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:48.798858 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p6zz5_2f394829-084a-4947-8ac4-699f1b9708fd/serve-healthcheck-canary/0.log" Jul 24 17:21:49.599528 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:49.599492 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5_f6fd8a29-1f40-40e1-a279-21e27465afc4/kube-storage-version-migrator-operator/0.log" Jul 24 17:21:51.936336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:51.936297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:51.936799 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:51.936375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:21:51.936799 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:51.936475 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:21:51.936799 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:51.936509 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:21:51.936799 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:51.936545 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:59.936525739 +0000 UTC m=+276.602100368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : secret "insights-runtime-extractor-tls" not found Jul 24 17:21:51.936799 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:51.936564 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm podName:e3cd09a0-7a43-49c1-b7ba-afe6afdb9181 nodeName:}" failed. No retries permitted until 2026-07-24 17:21:59.936555483 +0000 UTC m=+276.602130103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5png" (UID: "e3cd09a0-7a43-49c1-b7ba-afe6afdb9181") : configmap "kube-rbac-proxy" not found Jul 24 17:21:54.255917 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:54.255885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:54.256315 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:54.255942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:54.256315 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:54.255999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:21:54.256315 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:54.256039 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle podName:4b6dc3ec-0cb7-4b72-bbd6-58922b496519 nodeName:}" failed. No retries permitted until 2026-07-24 17:22:10.256016879 +0000 UTC m=+286.921591494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle") pod "router-default-67844487c7-fg86z" (UID: "4b6dc3ec-0cb7-4b72-bbd6-58922b496519") : configmap references non-existent config key: service-ca.crt Jul 24 17:21:54.256315 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:54.256107 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:54.256315 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:21:54.256162 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls podName:81f9cda0-d765-4821-b23a-62d5a9f17b12 nodeName:}" failed. No retries permitted until 2026-07-24 17:22:10.256149048 +0000 UTC m=+286.921723665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ngcgm" (UID: "81f9cda0-d765-4821-b23a-62d5a9f17b12") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:21:54.258514 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:54.258493 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-metrics-certs\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:21:59.795994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:21:59.795958 2574 scope.go:117] "RemoveContainer" containerID="f352a5cb988befad46612cbab3016037c0f6563f6448ed752e82aec09f85dc8e" Jul 24 17:22:00.002510 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.002477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:22:00.002698 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.002553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:22:00.003165 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.003134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:22:00.005070 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.005046 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3cd09a0-7a43-49c1-b7ba-afe6afdb9181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5png\" (UID: \"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181\") " pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:22:00.123029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.122945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m5png" Jul 24 17:22:00.258657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.258628 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m5png"] Jul 24 17:22:00.261849 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:00.261823 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cd09a0_7a43_49c1_b7ba_afe6afdb9181.slice/crio-fa794e7fd79cab1c101a16f7bc52d4563e0435b315a3fecee11a26f4a33c67da WatchSource:0}: Error finding container fa794e7fd79cab1c101a16f7bc52d4563e0435b315a3fecee11a26f4a33c67da: Status 404 returned error can't find the container with id fa794e7fd79cab1c101a16f7bc52d4563e0435b315a3fecee11a26f4a33c67da Jul 24 17:22:00.751077 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.751045 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5png" event={"ID":"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181","Type":"ContainerStarted","Data":"fa794e7fd79cab1c101a16f7bc52d4563e0435b315a3fecee11a26f4a33c67da"} Jul 24 17:22:00.752389 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.752369 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:22:00.752806 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.752790 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/1.log" Jul 24 17:22:00.752902 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.752826 2574 generic.go:356] "Generic (PLEG): container finished" podID="411738aa-268d-4f8d-b8bd-1f8cfc0ebff4" containerID="c53ac82e1440c2917f22b16d4c36992582f0e7b45e8c29aef7fdc709e02f5446" exitCode=255 Jul 24 17:22:00.752902 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.752855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" event={"ID":"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4","Type":"ContainerDied","Data":"c53ac82e1440c2917f22b16d4c36992582f0e7b45e8c29aef7fdc709e02f5446"} Jul 24 17:22:00.753013 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.752909 2574 scope.go:117] "RemoveContainer" containerID="f352a5cb988befad46612cbab3016037c0f6563f6448ed752e82aec09f85dc8e" Jul 24 17:22:00.753263 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:00.753249 2574 scope.go:117] "RemoveContainer" containerID="c53ac82e1440c2917f22b16d4c36992582f0e7b45e8c29aef7fdc709e02f5446" Jul 24 17:22:00.753451 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:22:00.753424 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-64lhm_openshift-console-operator(411738aa-268d-4f8d-b8bd-1f8cfc0ebff4)\"" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" podUID="411738aa-268d-4f8d-b8bd-1f8cfc0ebff4" Jul 24 17:22:01.757625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:01.757597 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:22:05.437695 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:22:05.434425 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dbmj4" podUID="ff664f7f-5d45-4149-8c35-4d70dbc8d401" Jul 24 17:22:05.437695 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:22:05.434523 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" podUID="99419e6a-9ace-4182-aedb-c4f0b350a36c" Jul 24 17:22:05.776990 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:05.776958 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbmj4" Jul 24 17:22:05.776990 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:05.776964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5png" event={"ID":"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181","Type":"ContainerStarted","Data":"dcdf32ebbddab25bb0dfaa7f19006b3f8af151a10a350a27c463690b7a92aa4d"} Jul 24 17:22:05.777288 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:05.776997 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:22:08.667972 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:08.667936 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:22:08.668331 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:08.667997 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:22:08.668331 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:08.668314 2574 scope.go:117] "RemoveContainer" containerID="c53ac82e1440c2917f22b16d4c36992582f0e7b45e8c29aef7fdc709e02f5446" Jul 24 17:22:08.668490 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:22:08.668469 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-64lhm_openshift-console-operator(411738aa-268d-4f8d-b8bd-1f8cfc0ebff4)\"" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" podUID="411738aa-268d-4f8d-b8bd-1f8cfc0ebff4" Jul 24 17:22:08.788940 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:08.788851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5png" event={"ID":"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181","Type":"ContainerStarted","Data":"028200c6f0dd85e097c8fd09f28da35706189f1b1b650b86fceb1e078fc6298d"} Jul 24 17:22:09.185323 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.185231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:22:09.185323 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.185279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:22:09.187947 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.187917 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff664f7f-5d45-4149-8c35-4d70dbc8d401-metrics-tls\") pod \"dns-default-dbmj4\" (UID: \"ff664f7f-5d45-4149-8c35-4d70dbc8d401\") " pod="openshift-dns/dns-default-dbmj4" Jul 24 17:22:09.188161 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.188140 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"image-registry-6fcdc7c484-ccpzp\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:22:09.382421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.382388 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p7fvc\"" Jul 24 17:22:09.382421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.382389 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2b86\"" Jul 24 17:22:09.389035 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.389016 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbmj4" Jul 24 17:22:09.389201 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.389185 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:22:09.554960 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.554928 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fcdc7c484-ccpzp"] Jul 24 17:22:09.558717 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:09.558686 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99419e6a_9ace_4182_aedb_c4f0b350a36c.slice/crio-c93b76bfedf4cc6f5c4b4ee739c20437e6a2f08305ece2fe2c92a1642d81981d WatchSource:0}: Error finding container c93b76bfedf4cc6f5c4b4ee739c20437e6a2f08305ece2fe2c92a1642d81981d: Status 404 returned error can't find the container with id c93b76bfedf4cc6f5c4b4ee739c20437e6a2f08305ece2fe2c92a1642d81981d Jul 24 17:22:09.581052 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.581031 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbmj4"] Jul 24 17:22:09.590181 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:09.590146 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff664f7f_5d45_4149_8c35_4d70dbc8d401.slice/crio-1bd6548abb45543bf234807bba77fb1ddf66eca24cf096eed1e37bf16d42510a WatchSource:0}: Error finding container 1bd6548abb45543bf234807bba77fb1ddf66eca24cf096eed1e37bf16d42510a: Status 404 returned error can't find the container with id 1bd6548abb45543bf234807bba77fb1ddf66eca24cf096eed1e37bf16d42510a Jul 24 17:22:09.793632 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.793595 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbmj4" event={"ID":"ff664f7f-5d45-4149-8c35-4d70dbc8d401","Type":"ContainerStarted","Data":"1bd6548abb45543bf234807bba77fb1ddf66eca24cf096eed1e37bf16d42510a"} Jul 24 17:22:09.799353 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.799324 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:22:09.799493 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.799379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" event={"ID":"99419e6a-9ace-4182-aedb-c4f0b350a36c","Type":"ContainerStarted","Data":"a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259"} Jul 24 17:22:09.799968 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.799747 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" event={"ID":"99419e6a-9ace-4182-aedb-c4f0b350a36c","Type":"ContainerStarted","Data":"c93b76bfedf4cc6f5c4b4ee739c20437e6a2f08305ece2fe2c92a1642d81981d"} Jul 24 17:22:09.815997 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:09.815925 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" podStartSLOduration=290.815905775 podStartE2EDuration="4m50.815905775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:09.814983226 +0000 UTC m=+286.480557865" watchObservedRunningTime="2026-07-24 17:22:09.815905775 +0000 UTC m=+286.481480413" Jul 24 17:22:10.299047 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:10.298430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:22:10.299047 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:10.298542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:22:10.299447 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:10.299395 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b6dc3ec-0cb7-4b72-bbd6-58922b496519-service-ca-bundle\") pod \"router-default-67844487c7-fg86z\" (UID: \"4b6dc3ec-0cb7-4b72-bbd6-58922b496519\") " pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:22:10.301485 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:10.301463 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81f9cda0-d765-4821-b23a-62d5a9f17b12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ngcgm\" (UID: \"81f9cda0-d765-4821-b23a-62d5a9f17b12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:22:10.407952 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:10.407914 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcdc7c484-ccpzp"] Jul 24 17:22:10.472486 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:10.472250 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:22:10.542445 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:10.542384 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" Jul 24 17:22:12.750108 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.750076 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67844487c7-fg86z"] Jul 24 17:22:12.755700 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:12.755120 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6dc3ec_0cb7_4b72_bbd6_58922b496519.slice/crio-585574d876c04ccb95d3485c9793ef6d8a32b882191220c166781896e88d0554 WatchSource:0}: Error finding container 585574d876c04ccb95d3485c9793ef6d8a32b882191220c166781896e88d0554: Status 404 returned error can't find the container with id 585574d876c04ccb95d3485c9793ef6d8a32b882191220c166781896e88d0554 Jul 24 17:22:12.765476 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.765455 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm"] Jul 24 17:22:12.770769 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:12.770747 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f9cda0_d765_4821_b23a_62d5a9f17b12.slice/crio-4ba3284588c7ad40bb962af769e6f3684dbb1ebd838d7b46aac23c2052f59293 WatchSource:0}: Error finding container 4ba3284588c7ad40bb962af769e6f3684dbb1ebd838d7b46aac23c2052f59293: Status 404 returned error can't find the container with id 4ba3284588c7ad40bb962af769e6f3684dbb1ebd838d7b46aac23c2052f59293 Jul 24 17:22:12.807593 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.807351 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbmj4" event={"ID":"ff664f7f-5d45-4149-8c35-4d70dbc8d401","Type":"ContainerStarted","Data":"ff0916b66f2faa381ede908382f24a08980fbb6c1ac693272e0cccf3e4caddf5"} Jul 24 17:22:12.807593 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.807416 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbmj4" event={"ID":"ff664f7f-5d45-4149-8c35-4d70dbc8d401","Type":"ContainerStarted","Data":"8730965e344b17d0fbad1f9529726ef05ae1bae23269ed7d6ab8773bc08b8dab"} Jul 24 17:22:12.807593 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.807536 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dbmj4" Jul 24 17:22:12.809206 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.809170 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" event={"ID":"81f9cda0-d765-4821-b23a-62d5a9f17b12","Type":"ContainerStarted","Data":"4ba3284588c7ad40bb962af769e6f3684dbb1ebd838d7b46aac23c2052f59293"} Jul 24 17:22:12.811830 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.811764 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67844487c7-fg86z" event={"ID":"4b6dc3ec-0cb7-4b72-bbd6-58922b496519","Type":"ContainerStarted","Data":"b9b31a6e93820f7a7e5a4a7e17ca2ee2e3c470ec07d766387177be158ac8b7e0"} Jul 24 17:22:12.811830 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.811804 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67844487c7-fg86z" event={"ID":"4b6dc3ec-0cb7-4b72-bbd6-58922b496519","Type":"ContainerStarted","Data":"585574d876c04ccb95d3485c9793ef6d8a32b882191220c166781896e88d0554"} Jul 24 17:22:12.814152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.814097 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5png" event={"ID":"e3cd09a0-7a43-49c1-b7ba-afe6afdb9181","Type":"ContainerStarted","Data":"4fbf3fbd381398d840b1c5784fae1522683a290f73e84a0f89a43b1f8b482255"} Jul 24 17:22:12.826508 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.826457 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dbmj4" podStartSLOduration=250.814852592 podStartE2EDuration="4m13.826440341s" totalImagesPullingTime="3.011587749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:12.824045428 +0000 UTC m=+289.489620068" watchObservedRunningTime="2026-07-24 17:22:12.826440341 +0000 UTC m=+289.492014979" Jul 24 17:22:12.844445 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.844388 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-67844487c7-fg86z" podStartSLOduration=34.844372122 podStartE2EDuration="34.844372122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:12.843008408 +0000 UTC m=+289.508583047" watchObservedRunningTime="2026-07-24 17:22:12.844372122 +0000 UTC m=+289.509946760" Jul 24 17:22:12.862674 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:12.862596 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m5png" podStartSLOduration=16.63484214 podStartE2EDuration="28.862580703s" totalImagesPullingTime="12.227738563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:21:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:12.861709587 +0000 UTC m=+289.527284224" watchObservedRunningTime="2026-07-24 17:22:12.862580703 +0000 UTC m=+289.528155341" Jul 24 17:22:13.473279 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:13.473244 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:22:13.476933 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:13.476909 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:22:13.817839 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:13.817758 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:22:13.819016 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:13.818993 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-67844487c7-fg86z" Jul 24 17:22:14.821838 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:14.821757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" event={"ID":"81f9cda0-d765-4821-b23a-62d5a9f17b12","Type":"ContainerStarted","Data":"545136f4047693220a7250d12a79061481beb512d800af9d49ffb2dd3d5f390a"} Jul 24 17:22:14.843027 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:14.842983 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ngcgm" podStartSLOduration=35.108392592 podStartE2EDuration="36.842966837s" totalImagesPullingTime="1.734574245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:14.841904037 +0000 UTC m=+291.507478688" watchObservedRunningTime="2026-07-24 17:22:14.842966837 +0000 UTC m=+291.508541474" Jul 24 17:22:18.047386 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.047348 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-z5h9c"] Jul 24 17:22:18.050369 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.050353 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.054956 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.054929 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 17:22:18.055094 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.054932 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 24 17:22:18.055094 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.054935 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7kv5v\"" Jul 24 17:22:18.055094 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.054963 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 24 17:22:18.058338 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.058314 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-z5h9c"] Jul 24 17:22:18.167951 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.167918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b48472-265f-4667-8191-6ff1e5b316fe-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.168125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.167990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8b48472-265f-4667-8191-6ff1e5b316fe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.168125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.168035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gb6\" (UniqueName: \"kubernetes.io/projected/a8b48472-265f-4667-8191-6ff1e5b316fe-kube-api-access-z7gb6\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.168125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.168064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8b48472-265f-4667-8191-6ff1e5b316fe-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.269344 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.269254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gb6\" (UniqueName: \"kubernetes.io/projected/a8b48472-265f-4667-8191-6ff1e5b316fe-kube-api-access-z7gb6\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.269542 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.269381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8b48472-265f-4667-8191-6ff1e5b316fe-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.269542 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.269444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b48472-265f-4667-8191-6ff1e5b316fe-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.269542 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.269506 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8b48472-265f-4667-8191-6ff1e5b316fe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.270145 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.270121 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b48472-265f-4667-8191-6ff1e5b316fe-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.271860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.271838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8b48472-265f-4667-8191-6ff1e5b316fe-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.271946 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.271859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8b48472-265f-4667-8191-6ff1e5b316fe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.277602 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.277582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gb6\" (UniqueName: \"kubernetes.io/projected/a8b48472-265f-4667-8191-6ff1e5b316fe-kube-api-access-z7gb6\") pod \"prometheus-operator-84d69f794d-z5h9c\" (UID: \"a8b48472-265f-4667-8191-6ff1e5b316fe\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.360618 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.360521 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" Jul 24 17:22:18.488171 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.488140 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-z5h9c"] Jul 24 17:22:18.491345 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:18.491317 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b48472_265f_4667_8191_6ff1e5b316fe.slice/crio-f4c7f41980e226daca4f5f6c16f8f220664ed581ca9bfa2b83e8a9a83adab791 WatchSource:0}: Error finding container f4c7f41980e226daca4f5f6c16f8f220664ed581ca9bfa2b83e8a9a83adab791: Status 404 returned error can't find the container with id f4c7f41980e226daca4f5f6c16f8f220664ed581ca9bfa2b83e8a9a83adab791 Jul 24 17:22:18.836284 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:18.836247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" event={"ID":"a8b48472-265f-4667-8191-6ff1e5b316fe","Type":"ContainerStarted","Data":"f4c7f41980e226daca4f5f6c16f8f220664ed581ca9bfa2b83e8a9a83adab791"} Jul 24 17:22:20.795744 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:22:20.795704 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6t9nt" podUID="bd39d23d-0f58-49d6-8d42-39caa7c31c41" Jul 24 17:22:20.845283 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:20.845247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" event={"ID":"a8b48472-265f-4667-8191-6ff1e5b316fe","Type":"ContainerStarted","Data":"453d1295323b63a2a35f5e6931c5719257fecbe67f5ce700869e89501089f5be"} Jul 24 17:22:20.845425 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:20.845298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" event={"ID":"a8b48472-265f-4667-8191-6ff1e5b316fe","Type":"ContainerStarted","Data":"7ef4dd6b03faa0bedcad4c8a32a1770658e486a1e645f9fff57ec6c609378131"} Jul 24 17:22:20.862984 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:20.862939 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-z5h9c" podStartSLOduration=1.428016884 podStartE2EDuration="2.862924607s" totalImagesPullingTime="1.434907723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:20.861770401 +0000 UTC m=+297.527345042" watchObservedRunningTime="2026-07-24 17:22:20.862924607 +0000 UTC m=+297.528499244" Jul 24 17:22:22.394241 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.394204 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7"] Jul 24 17:22:22.396985 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.396967 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.400115 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.400093 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rwt86\"" Jul 24 17:22:22.400224 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.400210 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 24 17:22:22.400289 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.400093 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 24 17:22:22.409282 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.409261 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7"] Jul 24 17:22:22.437705 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.437682 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dnf9v"] Jul 24 17:22:22.439849 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.439835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.442996 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.442791 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 17:22:22.442996 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.442848 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 17:22:22.442996 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.442919 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 17:22:22.443223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.443100 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2zddw\"" Jul 24 17:22:22.506431 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.506396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a6108b7-073e-4200-abe3-88d4f5758ffb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.506566 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.506443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6108b7-073e-4200-abe3-88d4f5758ffb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.506566 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.506519 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a6108b7-073e-4200-abe3-88d4f5758ffb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.506566 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.506546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2tb7\" (UniqueName: \"kubernetes.io/projected/7a6108b7-073e-4200-abe3-88d4f5758ffb-kube-api-access-p2tb7\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.607273 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607238 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354d1d50-0c63-45c4-b252-620ca605b4ef-metrics-client-ca\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.607458 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-wtmp\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.607458 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a6108b7-073e-4200-abe3-88d4f5758ffb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.607458 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2tb7\" (UniqueName: \"kubernetes.io/projected/7a6108b7-073e-4200-abe3-88d4f5758ffb-kube-api-access-p2tb7\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.607458 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/354d1d50-0c63-45c4-b252-620ca605b4ef-sys\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.607458 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.607458 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf24m\" (UniqueName: \"kubernetes.io/projected/354d1d50-0c63-45c4-b252-620ca605b4ef-kube-api-access-lf24m\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.607706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607483 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-tls\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.607706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.607706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/354d1d50-0c63-45c4-b252-620ca605b4ef-root\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.607706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a6108b7-073e-4200-abe3-88d4f5758ffb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.607706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-textfile\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.607706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.607672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6108b7-073e-4200-abe3-88d4f5758ffb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.608304 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.608274 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a6108b7-073e-4200-abe3-88d4f5758ffb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.609982 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.609962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a6108b7-073e-4200-abe3-88d4f5758ffb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.610184 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.610166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a6108b7-073e-4200-abe3-88d4f5758ffb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.616867 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.616848 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2tb7\" (UniqueName: \"kubernetes.io/projected/7a6108b7-073e-4200-abe3-88d4f5758ffb-kube-api-access-p2tb7\") pod \"openshift-state-metrics-b8f49f5c6-tcxg7\" (UID: \"7a6108b7-073e-4200-abe3-88d4f5758ffb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.708218 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" Jul 24 17:22:22.708446 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-wtmp\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708513 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/354d1d50-0c63-45c4-b252-620ca605b4ef-sys\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708552 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708552 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf24m\" (UniqueName: \"kubernetes.io/projected/354d1d50-0c63-45c4-b252-620ca605b4ef-kube-api-access-lf24m\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708646 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-tls\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708646 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708598 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-wtmp\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708646 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/354d1d50-0c63-45c4-b252-620ca605b4ef-sys\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708646 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708856 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/354d1d50-0c63-45c4-b252-620ca605b4ef-root\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708856 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-textfile\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708856 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708717 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354d1d50-0c63-45c4-b252-620ca605b4ef-metrics-client-ca\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.708856 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.708776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/354d1d50-0c63-45c4-b252-620ca605b4ef-root\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.709083 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.709061 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-textfile\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.709263 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.709238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.709398 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.709247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354d1d50-0c63-45c4-b252-620ca605b4ef-metrics-client-ca\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.711608 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.711586 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.711971 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.711948 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/354d1d50-0c63-45c4-b252-620ca605b4ef-node-exporter-tls\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.716915 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.716891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf24m\" (UniqueName: \"kubernetes.io/projected/354d1d50-0c63-45c4-b252-620ca605b4ef-kube-api-access-lf24m\") pod \"node-exporter-dnf9v\" (UID: \"354d1d50-0c63-45c4-b252-620ca605b4ef\") " pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.749087 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.749054 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dnf9v" Jul 24 17:22:22.762917 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:22.762881 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354d1d50_0c63_45c4_b252_620ca605b4ef.slice/crio-9827692c64e5495618b935ce711f661eaf88014d17b106d60578c9b133641da7 WatchSource:0}: Error finding container 9827692c64e5495618b935ce711f661eaf88014d17b106d60578c9b133641da7: Status 404 returned error can't find the container with id 9827692c64e5495618b935ce711f661eaf88014d17b106d60578c9b133641da7 Jul 24 17:22:22.819120 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.819093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbmj4" Jul 24 17:22:22.853089 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.853060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dnf9v" event={"ID":"354d1d50-0c63-45c4-b252-620ca605b4ef","Type":"ContainerStarted","Data":"9827692c64e5495618b935ce711f661eaf88014d17b106d60578c9b133641da7"} Jul 24 17:22:22.863171 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:22.863149 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7"] Jul 24 17:22:22.866741 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:22.866698 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6108b7_073e_4200_abe3_88d4f5758ffb.slice/crio-52220fd5031083790dc43c45d276799a57cd87a683e56cfdf6d034045ce83911 WatchSource:0}: Error finding container 52220fd5031083790dc43c45d276799a57cd87a683e56cfdf6d034045ce83911: Status 404 returned error can't find the container with id 52220fd5031083790dc43c45d276799a57cd87a683e56cfdf6d034045ce83911 Jul 24 17:22:23.692092 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:23.692070 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:22:23.693151 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:23.693132 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:22:23.703378 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:23.703359 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 17:22:23.797747 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:23.797589 2574 scope.go:117] "RemoveContainer" containerID="c53ac82e1440c2917f22b16d4c36992582f0e7b45e8c29aef7fdc709e02f5446" Jul 24 17:22:23.860407 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:23.860282 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" event={"ID":"7a6108b7-073e-4200-abe3-88d4f5758ffb","Type":"ContainerStarted","Data":"afc1aa4b4b8a3109c369b815f0278e11428052ca55fe05fe071bb98e58218119"} Jul 24 17:22:23.860407 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:23.860348 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" event={"ID":"7a6108b7-073e-4200-abe3-88d4f5758ffb","Type":"ContainerStarted","Data":"94be336fa8ad70b0a6ea9a186ea8022e3feb7f9d32321e8aa9844bc412454319"} Jul 24 17:22:23.860407 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:23.860375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" event={"ID":"7a6108b7-073e-4200-abe3-88d4f5758ffb","Type":"ContainerStarted","Data":"52220fd5031083790dc43c45d276799a57cd87a683e56cfdf6d034045ce83911"} Jul 24 17:22:23.862157 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:23.862128 2574 generic.go:356] "Generic (PLEG): container finished" podID="354d1d50-0c63-45c4-b252-620ca605b4ef" containerID="4f4528ce031dc37aef2e3665d8075cb4953d205130be7848ae8a879755ab3913" exitCode=0 Jul 24 17:22:23.862255 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:23.862160 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dnf9v" event={"ID":"354d1d50-0c63-45c4-b252-620ca605b4ef","Type":"ContainerDied","Data":"4f4528ce031dc37aef2e3665d8075cb4953d205130be7848ae8a879755ab3913"} Jul 24 17:22:24.868634 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.868608 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:22:24.869045 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.868744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" event={"ID":"411738aa-268d-4f8d-b8bd-1f8cfc0ebff4","Type":"ContainerStarted","Data":"50266fa138382a871b8d54ce310ea36ec8140461503e17cb6f34dbb3f1adf598"} Jul 24 17:22:24.869096 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.869059 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:22:24.871158 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.871116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" event={"ID":"7a6108b7-073e-4200-abe3-88d4f5758ffb","Type":"ContainerStarted","Data":"e5a7900b9f81dcbe37d738177829ba323af05e0911a803a66ff28dc411b09431"} Jul 24 17:22:24.873541 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.873514 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dnf9v" event={"ID":"354d1d50-0c63-45c4-b252-620ca605b4ef","Type":"ContainerStarted","Data":"4baaf2e5a71b5a2faf441b0c95772a8bcac8531a2f1819d9ce69b0095d4c74a8"} Jul 24 17:22:24.873673 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.873552 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dnf9v" event={"ID":"354d1d50-0c63-45c4-b252-620ca605b4ef","Type":"ContainerStarted","Data":"c62013fbf4b2d5a32cabe0581a5580d45a59b05bd2f08960f1c3eb807c47d239"} Jul 24 17:22:24.874500 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.874481 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" Jul 24 17:22:24.889451 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.889407 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-64lhm" podStartSLOduration=43.284785593 podStartE2EDuration="46.889394848s" totalImagesPullingTime="3.604609255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:24.888235288 +0000 UTC m=+301.553809934" watchObservedRunningTime="2026-07-24 17:22:24.889394848 +0000 UTC m=+301.554969485" Jul 24 17:22:24.910127 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.910087 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-tcxg7" podStartSLOduration=1.5990492299999999 podStartE2EDuration="2.910075096s" totalImagesPullingTime="1.311025866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:24.909213458 +0000 UTC m=+301.574788095" watchObservedRunningTime="2026-07-24 17:22:24.910075096 +0000 UTC m=+301.575649733" Jul 24 17:22:24.953402 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:24.953347 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dnf9v" podStartSLOduration=1.95746623 podStartE2EDuration="2.953329844s" totalImagesPullingTime="913.318178ms" totalInitContainerRuntime="82.545436ms" isStatefulPod=true podCreationTimestamp="2026-07-24 17:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:24.952284443 +0000 UTC m=+301.617859082" watchObservedRunningTime="2026-07-24 17:22:24.953329844 +0000 UTC m=+301.618904527" Jul 24 17:22:31.807295 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:31.807269 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:22:34.570991 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.570961 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77867b7b8f-jrstp"] Jul 24 17:22:34.580422 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.580401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.585056 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.585032 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 17:22:34.585235 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.585196 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 17:22:34.585235 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.585126 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tpt4h\"" Jul 24 17:22:34.585350 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.585127 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 17:22:34.585350 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.585158 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 17:22:34.585350 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.585160 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 17:22:34.585350 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.585178 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 17:22:34.585559 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.585113 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 17:22:34.588504 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.588485 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77867b7b8f-jrstp"] Jul 24 17:22:34.603283 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.603261 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-oauth-config\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.603376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.603291 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-config\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.603376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.603313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-oauth-serving-cert\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.603453 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.603382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-serving-cert\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.603453 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.603417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-service-ca\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.603453 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.603446 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjqf\" (UniqueName: \"kubernetes.io/projected/2e909d6d-2a8f-46ab-8f37-e839513e692d-kube-api-access-xkjqf\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.704641 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.704614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-config\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.704815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.704655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-oauth-serving-cert\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.704815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.704694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-serving-cert\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.704815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.704750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-service-ca\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.704815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.704785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjqf\" (UniqueName: \"kubernetes.io/projected/2e909d6d-2a8f-46ab-8f37-e839513e692d-kube-api-access-xkjqf\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.705015 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.704833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-oauth-config\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.705343 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.705311 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-config\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.705496 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.705341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-oauth-serving-cert\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.705496 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.705510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-service-ca\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.707372 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.707349 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-oauth-config\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.707372 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.707364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-serving-cert\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.713790 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.713770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjqf\" (UniqueName: \"kubernetes.io/projected/2e909d6d-2a8f-46ab-8f37-e839513e692d-kube-api-access-xkjqf\") pod \"console-77867b7b8f-jrstp\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:34.795069 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.795042 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:22:34.893506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:34.893397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:35.028213 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:35.028186 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77867b7b8f-jrstp"] Jul 24 17:22:35.030641 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:35.030602 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e909d6d_2a8f_46ab_8f37_e839513e692d.slice/crio-6778b73d109ed3bd7b2cf924e10b367360e311840d5b8068cce50f7094672373 WatchSource:0}: Error finding container 6778b73d109ed3bd7b2cf924e10b367360e311840d5b8068cce50f7094672373: Status 404 returned error can't find the container with id 6778b73d109ed3bd7b2cf924e10b367360e311840d5b8068cce50f7094672373 Jul 24 17:22:35.032605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:35.032589 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:22:35.919332 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:35.919288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77867b7b8f-jrstp" event={"ID":"2e909d6d-2a8f-46ab-8f37-e839513e692d","Type":"ContainerStarted","Data":"6778b73d109ed3bd7b2cf924e10b367360e311840d5b8068cce50f7094672373"} Jul 24 17:22:36.822488 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:36.822409 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" podUID="99419e6a-9ace-4182-aedb-c4f0b350a36c" containerName="registry" containerID="cri-o://a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259" gracePeriod=30 Jul 24 17:22:37.883857 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.883837 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:22:37.928644 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.928608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77867b7b8f-jrstp" event={"ID":"2e909d6d-2a8f-46ab-8f37-e839513e692d","Type":"ContainerStarted","Data":"cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1"} Jul 24 17:22:37.929995 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.929966 2574 generic.go:356] "Generic (PLEG): container finished" podID="99419e6a-9ace-4182-aedb-c4f0b350a36c" containerID="a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259" exitCode=0 Jul 24 17:22:37.930123 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.930010 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" Jul 24 17:22:37.930123 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.930035 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" event={"ID":"99419e6a-9ace-4182-aedb-c4f0b350a36c","Type":"ContainerDied","Data":"a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259"} Jul 24 17:22:37.930123 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.930072 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fcdc7c484-ccpzp" event={"ID":"99419e6a-9ace-4182-aedb-c4f0b350a36c","Type":"ContainerDied","Data":"c93b76bfedf4cc6f5c4b4ee739c20437e6a2f08305ece2fe2c92a1642d81981d"} Jul 24 17:22:37.930123 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.930094 2574 scope.go:117] "RemoveContainer" containerID="a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259" Jul 24 17:22:37.936306 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936276 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") pod \"99419e6a-9ace-4182-aedb-c4f0b350a36c\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " Jul 24 17:22:37.936425 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936328 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99419e6a-9ace-4182-aedb-c4f0b350a36c-ca-trust-extracted\") pod \"99419e6a-9ace-4182-aedb-c4f0b350a36c\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " Jul 24 17:22:37.936425 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936355 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-trusted-ca\") pod \"99419e6a-9ace-4182-aedb-c4f0b350a36c\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " Jul 24 17:22:37.936425 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936388 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-installation-pull-secrets\") pod \"99419e6a-9ace-4182-aedb-c4f0b350a36c\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " Jul 24 17:22:37.936578 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936432 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-certificates\") pod \"99419e6a-9ace-4182-aedb-c4f0b350a36c\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " Jul 24 17:22:37.936578 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936452 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-image-registry-private-configuration\") pod \"99419e6a-9ace-4182-aedb-c4f0b350a36c\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " Jul 24 17:22:37.936578 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936474 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv6wp\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-kube-api-access-wv6wp\") pod \"99419e6a-9ace-4182-aedb-c4f0b350a36c\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " Jul 24 17:22:37.936578 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-bound-sa-token\") pod \"99419e6a-9ace-4182-aedb-c4f0b350a36c\" (UID: \"99419e6a-9ace-4182-aedb-c4f0b350a36c\") " Jul 24 17:22:37.937535 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936830 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "99419e6a-9ace-4182-aedb-c4f0b350a36c" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:37.937535 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.936950 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "99419e6a-9ace-4182-aedb-c4f0b350a36c" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:22:37.940968 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.940876 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "99419e6a-9ace-4182-aedb-c4f0b350a36c" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:37.940968 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.940919 2574 scope.go:117] "RemoveContainer" containerID="a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259" Jul 24 17:22:37.941129 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.940980 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-kube-api-access-wv6wp" (OuterVolumeSpecName: "kube-api-access-wv6wp") pod "99419e6a-9ace-4182-aedb-c4f0b350a36c" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c"). InnerVolumeSpecName "kube-api-access-wv6wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:22:37.941129 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.941045 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "99419e6a-9ace-4182-aedb-c4f0b350a36c" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:22:37.941235 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.941148 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "99419e6a-9ace-4182-aedb-c4f0b350a36c" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:22:37.941235 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:22:37.941210 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259\": container with ID starting with a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259 not found: ID does not exist" containerID="a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259" Jul 24 17:22:37.941340 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.941246 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259"} err="failed to get container status \"a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259\": rpc error: code = NotFound desc = could not find container \"a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259\": container with ID starting with a154d4b21ff426aaeaf1651d8c2b3bc724a09591812b1acd014c7d2deeb8f259 not found: ID does not exist" Jul 24 17:22:37.941446 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.941427 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "99419e6a-9ace-4182-aedb-c4f0b350a36c" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:22:37.946674 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.946654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99419e6a-9ace-4182-aedb-c4f0b350a36c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "99419e6a-9ace-4182-aedb-c4f0b350a36c" (UID: "99419e6a-9ace-4182-aedb-c4f0b350a36c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:22:37.965863 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:37.965819 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77867b7b8f-jrstp" podStartSLOduration=1.1897405189999999 podStartE2EDuration="3.965803193s" totalImagesPullingTime="2.776062674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:37.964379496 +0000 UTC m=+314.629954134" watchObservedRunningTime="2026-07-24 17:22:37.965803193 +0000 UTC m=+314.631377831" Jul 24 17:22:38.037455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.037360 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-installation-pull-secrets\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:22:38.037455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.037389 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-certificates\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:22:38.037455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.037403 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99419e6a-9ace-4182-aedb-c4f0b350a36c-image-registry-private-configuration\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:22:38.037455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.037418 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv6wp\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-kube-api-access-wv6wp\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:22:38.037455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.037430 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-bound-sa-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:22:38.037455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.037439 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99419e6a-9ace-4182-aedb-c4f0b350a36c-registry-tls\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:22:38.037455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.037446 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99419e6a-9ace-4182-aedb-c4f0b350a36c-ca-trust-extracted\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:22:38.037455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.037454 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99419e6a-9ace-4182-aedb-c4f0b350a36c-trusted-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:22:38.193389 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.193354 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86c679949f-tbzdn"] Jul 24 17:22:38.193713 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.193697 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99419e6a-9ace-4182-aedb-c4f0b350a36c" containerName="registry" Jul 24 17:22:38.193817 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.193717 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="99419e6a-9ace-4182-aedb-c4f0b350a36c" containerName="registry" Jul 24 17:22:38.193873 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.193815 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="99419e6a-9ace-4182-aedb-c4f0b350a36c" containerName="registry" Jul 24 17:22:38.198745 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.198711 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.208283 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.208261 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 17:22:38.209522 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.209500 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86c679949f-tbzdn"] Jul 24 17:22:38.238456 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.238435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-console-config\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.238532 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.238472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-service-ca\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.238532 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.238503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-serving-cert\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.238610 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.238571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-oauth-serving-cert\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.238610 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.238605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-trusted-ca-bundle\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.238670 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.238619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndql\" (UniqueName: \"kubernetes.io/projected/2d3554b2-4e43-45b9-8c1d-413b98737390-kube-api-access-2ndql\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.238670 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.238660 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-oauth-config\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.253383 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.253361 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcdc7c484-ccpzp"] Jul 24 17:22:38.255455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.255433 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fcdc7c484-ccpzp"] Jul 24 17:22:38.340056 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.339990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-console-config\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.340056 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.340030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-service-ca\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.340203 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.340075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-serving-cert\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.340203 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.340109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-oauth-serving-cert\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.340203 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.340148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-trusted-ca-bundle\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.340203 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.340168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndql\" (UniqueName: \"kubernetes.io/projected/2d3554b2-4e43-45b9-8c1d-413b98737390-kube-api-access-2ndql\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.340375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.340216 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-oauth-config\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.340809 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.340788 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-console-config\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.340904 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.340853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-service-ca\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.340904 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.340859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-oauth-serving-cert\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.341027 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.341012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-trusted-ca-bundle\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.343161 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.343138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-serving-cert\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.343241 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.343160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-oauth-config\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.348520 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.348500 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndql\" (UniqueName: \"kubernetes.io/projected/2d3554b2-4e43-45b9-8c1d-413b98737390-kube-api-access-2ndql\") pod \"console-86c679949f-tbzdn\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.507815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.507779 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:38.656389 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.656359 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86c679949f-tbzdn"] Jul 24 17:22:38.659240 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:22:38.659215 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3554b2_4e43_45b9_8c1d_413b98737390.slice/crio-75e4d9be5855bcc8590dfd4c710e30902d9bf1e4975fc10b8ae3946c178fe427 WatchSource:0}: Error finding container 75e4d9be5855bcc8590dfd4c710e30902d9bf1e4975fc10b8ae3946c178fe427: Status 404 returned error can't find the container with id 75e4d9be5855bcc8590dfd4c710e30902d9bf1e4975fc10b8ae3946c178fe427 Jul 24 17:22:38.936041 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.935948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c679949f-tbzdn" event={"ID":"2d3554b2-4e43-45b9-8c1d-413b98737390","Type":"ContainerStarted","Data":"9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c"} Jul 24 17:22:38.936041 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.936003 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c679949f-tbzdn" event={"ID":"2d3554b2-4e43-45b9-8c1d-413b98737390","Type":"ContainerStarted","Data":"75e4d9be5855bcc8590dfd4c710e30902d9bf1e4975fc10b8ae3946c178fe427"} Jul 24 17:22:38.955825 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:38.955777 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86c679949f-tbzdn" podStartSLOduration=0.955757649 podStartE2EDuration="955.757649ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:38.955547524 +0000 UTC m=+315.621122161" watchObservedRunningTime="2026-07-24 17:22:38.955757649 +0000 UTC m=+315.621332286" Jul 24 17:22:39.799531 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:39.799494 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99419e6a-9ace-4182-aedb-c4f0b350a36c" path="/var/lib/kubelet/pods/99419e6a-9ace-4182-aedb-c4f0b350a36c/volumes" Jul 24 17:22:44.894474 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:44.894442 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:44.894899 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:44.894499 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:44.899111 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:44.899088 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:44.962277 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:44.962250 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:22:48.507929 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:48.507893 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:48.508400 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:48.507954 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:48.512711 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:48.512690 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:48.973692 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:48.973658 2574 generic.go:356] "Generic (PLEG): container finished" podID="10aa3400-8f79-4e7f-97bb-d6ca24d01430" containerID="69c96f092af258c95fde8e0585707a6dec251bdbf068ca2348a3238d99620a74" exitCode=0 Jul 24 17:22:48.973870 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:48.973755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-hzg25" event={"ID":"10aa3400-8f79-4e7f-97bb-d6ca24d01430","Type":"ContainerDied","Data":"69c96f092af258c95fde8e0585707a6dec251bdbf068ca2348a3238d99620a74"} Jul 24 17:22:48.974252 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:48.974224 2574 scope.go:117] "RemoveContainer" containerID="69c96f092af258c95fde8e0585707a6dec251bdbf068ca2348a3238d99620a74" Jul 24 17:22:48.979747 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:48.979326 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:22:49.048880 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:49.048857 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77867b7b8f-jrstp"] Jul 24 17:22:49.980181 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:49.980143 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-hzg25" event={"ID":"10aa3400-8f79-4e7f-97bb-d6ca24d01430","Type":"ContainerStarted","Data":"a627ce5adff2ce4d811cbacaeaf826def62aeb1c782495383114121f6093b214"} Jul 24 17:22:51.182872 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:51.182849 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-ngcgm_81f9cda0-d765-4821-b23a-62d5a9f17b12/cluster-monitoring-operator/0.log" Jul 24 17:22:52.381029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:52.381003 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dnf9v_354d1d50-0c63-45c4-b252-620ca605b4ef/init-textfile/0.log" Jul 24 17:22:52.582324 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:52.582293 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dnf9v_354d1d50-0c63-45c4-b252-620ca605b4ef/node-exporter/0.log" Jul 24 17:22:52.781495 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:52.781465 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dnf9v_354d1d50-0c63-45c4-b252-620ca605b4ef/kube-rbac-proxy/0.log" Jul 24 17:22:54.181793 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:54.181765 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-tcxg7_7a6108b7-073e-4200-abe3-88d4f5758ffb/kube-rbac-proxy-main/0.log" Jul 24 17:22:54.381675 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:54.381647 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-tcxg7_7a6108b7-073e-4200-abe3-88d4f5758ffb/kube-rbac-proxy-self/0.log" Jul 24 17:22:54.582012 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:54.581984 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-tcxg7_7a6108b7-073e-4200-abe3-88d4f5758ffb/openshift-state-metrics/0.log" Jul 24 17:22:56.183045 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:56.183017 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-z5h9c_a8b48472-265f-4667-8191-6ff1e5b316fe/prometheus-operator/0.log" Jul 24 17:22:56.381683 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:56.381657 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-z5h9c_a8b48472-265f-4667-8191-6ff1e5b316fe/kube-rbac-proxy/0.log" Jul 24 17:22:58.781788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:58.781756 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:22:58.983555 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:58.983531 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/3.log" Jul 24 17:22:59.019490 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:59.019457 2574 generic.go:356] "Generic (PLEG): container finished" podID="a3a496af-b330-41c0-a0e8-acbb1d153492" containerID="f0c98e156f9edb19c84f7542adfabb0855860793c9103612b839445e321c7231" exitCode=0 Jul 24 17:22:59.019650 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:59.019527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" event={"ID":"a3a496af-b330-41c0-a0e8-acbb1d153492","Type":"ContainerDied","Data":"f0c98e156f9edb19c84f7542adfabb0855860793c9103612b839445e321c7231"} Jul 24 17:22:59.019896 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:59.019876 2574 scope.go:117] "RemoveContainer" containerID="f0c98e156f9edb19c84f7542adfabb0855860793c9103612b839445e321c7231" Jul 24 17:22:59.182090 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:59.182058 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77867b7b8f-jrstp_2e909d6d-2a8f-46ab-8f37-e839513e692d/console/0.log" Jul 24 17:22:59.383900 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:22:59.383833 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-86c679949f-tbzdn_2d3554b2-4e43-45b9-8c1d-413b98737390/console/0.log" Jul 24 17:23:00.024783 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:00.024751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9k69m" event={"ID":"a3a496af-b330-41c0-a0e8-acbb1d153492","Type":"ContainerStarted","Data":"7f95bb851b1548d8f019e94f3ec609d3d11a4e130db1ff20537d44dd3fa488ca"} Jul 24 17:23:03.038533 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:03.038500 2574 generic.go:356] "Generic (PLEG): container finished" podID="f6fd8a29-1f40-40e1-a279-21e27465afc4" containerID="e414d271af757f462f71437fc6ee64c4fe49a19e39689171f674ae5f47df9f6c" exitCode=0 Jul 24 17:23:03.039015 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:03.038553 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" event={"ID":"f6fd8a29-1f40-40e1-a279-21e27465afc4","Type":"ContainerDied","Data":"e414d271af757f462f71437fc6ee64c4fe49a19e39689171f674ae5f47df9f6c"} Jul 24 17:23:03.039015 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:03.038883 2574 scope.go:117] "RemoveContainer" containerID="e414d271af757f462f71437fc6ee64c4fe49a19e39689171f674ae5f47df9f6c" Jul 24 17:23:04.044333 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:04.044300 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5" event={"ID":"f6fd8a29-1f40-40e1-a279-21e27465afc4","Type":"ContainerStarted","Data":"c13379be8de2b3c39f46075256077d3aa841d9948e7a23987d2a96e645efda98"} Jul 24 17:23:14.069938 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.069854 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77867b7b8f-jrstp" podUID="2e909d6d-2a8f-46ab-8f37-e839513e692d" containerName="console" containerID="cri-o://cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1" gracePeriod=15 Jul 24 17:23:14.307483 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.307458 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:23:14.444130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.444043 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkjqf\" (UniqueName: \"kubernetes.io/projected/2e909d6d-2a8f-46ab-8f37-e839513e692d-kube-api-access-xkjqf\") pod \"2e909d6d-2a8f-46ab-8f37-e839513e692d\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " Jul 24 17:23:14.444130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.444080 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-oauth-config\") pod \"2e909d6d-2a8f-46ab-8f37-e839513e692d\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " Jul 24 17:23:14.444130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.444105 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-config\") pod \"2e909d6d-2a8f-46ab-8f37-e839513e692d\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " Jul 24 17:23:14.444405 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.444134 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-serving-cert\") pod \"2e909d6d-2a8f-46ab-8f37-e839513e692d\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " Jul 24 17:23:14.444405 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.444165 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-oauth-serving-cert\") pod \"2e909d6d-2a8f-46ab-8f37-e839513e692d\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " Jul 24 17:23:14.444405 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.444236 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-service-ca\") pod \"2e909d6d-2a8f-46ab-8f37-e839513e692d\" (UID: \"2e909d6d-2a8f-46ab-8f37-e839513e692d\") " Jul 24 17:23:14.444629 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.444583 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-config" (OuterVolumeSpecName: "console-config") pod "2e909d6d-2a8f-46ab-8f37-e839513e692d" (UID: "2e909d6d-2a8f-46ab-8f37-e839513e692d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:23:14.444629 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.444591 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2e909d6d-2a8f-46ab-8f37-e839513e692d" (UID: "2e909d6d-2a8f-46ab-8f37-e839513e692d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:23:14.444820 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.444652 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-service-ca" (OuterVolumeSpecName: "service-ca") pod "2e909d6d-2a8f-46ab-8f37-e839513e692d" (UID: "2e909d6d-2a8f-46ab-8f37-e839513e692d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:23:14.446545 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.446518 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e909d6d-2a8f-46ab-8f37-e839513e692d-kube-api-access-xkjqf" (OuterVolumeSpecName: "kube-api-access-xkjqf") pod "2e909d6d-2a8f-46ab-8f37-e839513e692d" (UID: "2e909d6d-2a8f-46ab-8f37-e839513e692d"). InnerVolumeSpecName "kube-api-access-xkjqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:23:14.446862 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.446839 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2e909d6d-2a8f-46ab-8f37-e839513e692d" (UID: "2e909d6d-2a8f-46ab-8f37-e839513e692d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:23:14.446921 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.446857 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2e909d6d-2a8f-46ab-8f37-e839513e692d" (UID: "2e909d6d-2a8f-46ab-8f37-e839513e692d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:23:14.545620 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.545588 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkjqf\" (UniqueName: \"kubernetes.io/projected/2e909d6d-2a8f-46ab-8f37-e839513e692d-kube-api-access-xkjqf\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:23:14.545620 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.545616 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-oauth-config\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:23:14.545620 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.545627 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-config\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:23:14.545944 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.545636 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e909d6d-2a8f-46ab-8f37-e839513e692d-console-serving-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:23:14.545944 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.545645 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-oauth-serving-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:23:14.545944 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:14.545653 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e909d6d-2a8f-46ab-8f37-e839513e692d-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:23:15.090894 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.090848 2574 generic.go:356] "Generic (PLEG): container finished" podID="2e909d6d-2a8f-46ab-8f37-e839513e692d" containerID="cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1" exitCode=0 Jul 24 17:23:15.091371 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.090909 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77867b7b8f-jrstp" Jul 24 17:23:15.091371 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.090917 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77867b7b8f-jrstp" event={"ID":"2e909d6d-2a8f-46ab-8f37-e839513e692d","Type":"ContainerDied","Data":"cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1"} Jul 24 17:23:15.091371 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.090993 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77867b7b8f-jrstp" event={"ID":"2e909d6d-2a8f-46ab-8f37-e839513e692d","Type":"ContainerDied","Data":"6778b73d109ed3bd7b2cf924e10b367360e311840d5b8068cce50f7094672373"} Jul 24 17:23:15.091371 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.091018 2574 scope.go:117] "RemoveContainer" containerID="cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1" Jul 24 17:23:15.100117 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.100099 2574 scope.go:117] "RemoveContainer" containerID="cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1" Jul 24 17:23:15.100385 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:23:15.100367 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1\": container with ID starting with cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1 not found: ID does not exist" containerID="cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1" Jul 24 17:23:15.100433 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.100394 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1"} err="failed to get container status \"cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1\": rpc error: code = NotFound desc = could not find container \"cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1\": container with ID starting with cf87dd3f0645713ba8e086c39dccf7ee74b8f12777470d9a62f81c336bef92d1 not found: ID does not exist" Jul 24 17:23:15.111888 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.111865 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77867b7b8f-jrstp"] Jul 24 17:23:15.114819 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.114794 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77867b7b8f-jrstp"] Jul 24 17:23:15.800156 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:15.800123 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e909d6d-2a8f-46ab-8f37-e839513e692d" path="/var/lib/kubelet/pods/2e909d6d-2a8f-46ab-8f37-e839513e692d/volumes" Jul 24 17:23:37.629077 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:37.629035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:23:37.631455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:37.631433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd39d23d-0f58-49d6-8d42-39caa7c31c41-metrics-certs\") pod \"network-metrics-daemon-6t9nt\" (UID: \"bd39d23d-0f58-49d6-8d42-39caa7c31c41\") " pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:23:37.798994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:37.798970 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ntfrz\"" Jul 24 17:23:37.805695 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:37.805676 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6t9nt" Jul 24 17:23:37.936451 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:37.936429 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6t9nt"] Jul 24 17:23:37.938472 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:23:37.938445 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd39d23d_0f58_49d6_8d42_39caa7c31c41.slice/crio-d6c334df5cd9611a8314203b93591b5ac772afef5b15e72ba10b2873727f6020 WatchSource:0}: Error finding container d6c334df5cd9611a8314203b93591b5ac772afef5b15e72ba10b2873727f6020: Status 404 returned error can't find the container with id d6c334df5cd9611a8314203b93591b5ac772afef5b15e72ba10b2873727f6020 Jul 24 17:23:38.175914 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:38.175821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6t9nt" event={"ID":"bd39d23d-0f58-49d6-8d42-39caa7c31c41","Type":"ContainerStarted","Data":"d6c334df5cd9611a8314203b93591b5ac772afef5b15e72ba10b2873727f6020"} Jul 24 17:23:40.188995 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:40.188962 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6t9nt" event={"ID":"bd39d23d-0f58-49d6-8d42-39caa7c31c41","Type":"ContainerStarted","Data":"bbbdd5cb0ccc7374eb554cf2854a731401e4376d75645935af30817d6b643d12"} Jul 24 17:23:40.189378 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:40.189036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6t9nt" event={"ID":"bd39d23d-0f58-49d6-8d42-39caa7c31c41","Type":"ContainerStarted","Data":"7c25647a2475b41ec90d075f0c6cf2e57a8cd23246eef5636a3cad2fa462db46"} Jul 24 17:23:40.206188 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:40.206141 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6t9nt" podStartSLOduration=374.977368523 podStartE2EDuration="6m16.206127015s" totalImagesPullingTime="1.228758492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:23:40.204959956 +0000 UTC m=+376.870534594" watchObservedRunningTime="2026-07-24 17:23:40.206127015 +0000 UTC m=+376.871701681" Jul 24 17:23:46.194918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.194887 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-654c7856b8-w5kfn"] Jul 24 17:23:46.195444 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.195178 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e909d6d-2a8f-46ab-8f37-e839513e692d" containerName="console" Jul 24 17:23:46.195444 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.195189 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e909d6d-2a8f-46ab-8f37-e839513e692d" containerName="console" Jul 24 17:23:46.195444 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.195254 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e909d6d-2a8f-46ab-8f37-e839513e692d" containerName="console" Jul 24 17:23:46.198330 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.198310 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.211266 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.211243 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654c7856b8-w5kfn"] Jul 24 17:23:46.299696 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.299660 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-console-config\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.299696 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.299701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-serving-cert\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.299928 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.299754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-service-ca\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.299928 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.299784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-trusted-ca-bundle\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.299928 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.299832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-oauth-config\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.299928 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.299855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgdn\" (UniqueName: \"kubernetes.io/projected/7497ef0a-c1a0-4df2-ac56-70564163d239-kube-api-access-2rgdn\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.299928 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.299893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-oauth-serving-cert\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.400746 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.400694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgdn\" (UniqueName: \"kubernetes.io/projected/7497ef0a-c1a0-4df2-ac56-70564163d239-kube-api-access-2rgdn\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.400918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.400776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-oauth-serving-cert\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.400918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.400809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-console-config\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.400918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.400844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-serving-cert\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.400918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.400864 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-service-ca\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.400918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.400886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-trusted-ca-bundle\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.401151 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.400948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-oauth-config\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.401611 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.401582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-oauth-serving-cert\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.401753 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.401644 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-service-ca\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.401815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.401745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-console-config\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.401935 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.401919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-trusted-ca-bundle\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.403409 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.403385 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-serving-cert\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.403492 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.403415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-oauth-config\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.408382 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.408362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgdn\" (UniqueName: \"kubernetes.io/projected/7497ef0a-c1a0-4df2-ac56-70564163d239-kube-api-access-2rgdn\") pod \"console-654c7856b8-w5kfn\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.511775 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.511720 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:46.647833 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:46.647810 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654c7856b8-w5kfn"] Jul 24 17:23:46.649832 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:23:46.649806 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7497ef0a_c1a0_4df2_ac56_70564163d239.slice/crio-deb6561d63fcde8ab861814d3529e3d808db8c9ab617160573fea6c597405383 WatchSource:0}: Error finding container deb6561d63fcde8ab861814d3529e3d808db8c9ab617160573fea6c597405383: Status 404 returned error can't find the container with id deb6561d63fcde8ab861814d3529e3d808db8c9ab617160573fea6c597405383 Jul 24 17:23:47.217917 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:47.217879 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654c7856b8-w5kfn" event={"ID":"7497ef0a-c1a0-4df2-ac56-70564163d239","Type":"ContainerStarted","Data":"9b939532a87dc0f8b8f6e837e5b8a696711945e573fc8cbb3631b43305543532"} Jul 24 17:23:47.218414 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:47.217931 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654c7856b8-w5kfn" event={"ID":"7497ef0a-c1a0-4df2-ac56-70564163d239","Type":"ContainerStarted","Data":"deb6561d63fcde8ab861814d3529e3d808db8c9ab617160573fea6c597405383"} Jul 24 17:23:47.240758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:47.240655 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-654c7856b8-w5kfn" podStartSLOduration=1.240635487 podStartE2EDuration="1.240635487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:23:47.239085807 +0000 UTC m=+383.904660445" watchObservedRunningTime="2026-07-24 17:23:47.240635487 +0000 UTC m=+383.906210125" Jul 24 17:23:56.512404 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:56.512372 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:56.512994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:56.512457 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:56.517099 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:56.517080 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:57.258677 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:57.258644 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:23:57.310212 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:23:57.310168 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86c679949f-tbzdn"] Jul 24 17:24:22.332096 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.332050 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86c679949f-tbzdn" podUID="2d3554b2-4e43-45b9-8c1d-413b98737390" containerName="console" containerID="cri-o://9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c" gracePeriod=15 Jul 24 17:24:22.581182 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.581159 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:24:22.590022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.589951 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-service-ca\") pod \"2d3554b2-4e43-45b9-8c1d-413b98737390\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " Jul 24 17:24:22.590022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.589988 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ndql\" (UniqueName: \"kubernetes.io/projected/2d3554b2-4e43-45b9-8c1d-413b98737390-kube-api-access-2ndql\") pod \"2d3554b2-4e43-45b9-8c1d-413b98737390\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " Jul 24 17:24:22.590022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.590012 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-serving-cert\") pod \"2d3554b2-4e43-45b9-8c1d-413b98737390\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " Jul 24 17:24:22.590226 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.590036 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-oauth-serving-cert\") pod \"2d3554b2-4e43-45b9-8c1d-413b98737390\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " Jul 24 17:24:22.590226 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.590068 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-oauth-config\") pod \"2d3554b2-4e43-45b9-8c1d-413b98737390\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " Jul 24 17:24:22.590226 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.590120 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-console-config\") pod \"2d3554b2-4e43-45b9-8c1d-413b98737390\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " Jul 24 17:24:22.590226 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.590161 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-trusted-ca-bundle\") pod \"2d3554b2-4e43-45b9-8c1d-413b98737390\" (UID: \"2d3554b2-4e43-45b9-8c1d-413b98737390\") " Jul 24 17:24:22.590461 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.590423 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-service-ca" (OuterVolumeSpecName: "service-ca") pod "2d3554b2-4e43-45b9-8c1d-413b98737390" (UID: "2d3554b2-4e43-45b9-8c1d-413b98737390"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:24:22.590645 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.590456 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2d3554b2-4e43-45b9-8c1d-413b98737390" (UID: "2d3554b2-4e43-45b9-8c1d-413b98737390"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:24:22.590645 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.590628 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-console-config" (OuterVolumeSpecName: "console-config") pod "2d3554b2-4e43-45b9-8c1d-413b98737390" (UID: "2d3554b2-4e43-45b9-8c1d-413b98737390"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:24:22.590783 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.590693 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2d3554b2-4e43-45b9-8c1d-413b98737390" (UID: "2d3554b2-4e43-45b9-8c1d-413b98737390"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:24:22.592351 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.592331 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2d3554b2-4e43-45b9-8c1d-413b98737390" (UID: "2d3554b2-4e43-45b9-8c1d-413b98737390"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:24:22.592810 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.592793 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3554b2-4e43-45b9-8c1d-413b98737390-kube-api-access-2ndql" (OuterVolumeSpecName: "kube-api-access-2ndql") pod "2d3554b2-4e43-45b9-8c1d-413b98737390" (UID: "2d3554b2-4e43-45b9-8c1d-413b98737390"). InnerVolumeSpecName "kube-api-access-2ndql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:24:22.592810 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.592796 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2d3554b2-4e43-45b9-8c1d-413b98737390" (UID: "2d3554b2-4e43-45b9-8c1d-413b98737390"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:24:22.691073 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.691034 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:24:22.691073 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.691066 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndql\" (UniqueName: \"kubernetes.io/projected/2d3554b2-4e43-45b9-8c1d-413b98737390-kube-api-access-2ndql\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:24:22.691073 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.691077 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-serving-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:24:22.691308 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.691087 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-oauth-serving-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:24:22.691308 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.691096 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d3554b2-4e43-45b9-8c1d-413b98737390-console-oauth-config\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:24:22.691308 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.691105 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-console-config\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:24:22.691308 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:22.691115 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d3554b2-4e43-45b9-8c1d-413b98737390-trusted-ca-bundle\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:24:23.348131 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.348100 2574 generic.go:356] "Generic (PLEG): container finished" podID="2d3554b2-4e43-45b9-8c1d-413b98737390" containerID="9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c" exitCode=0 Jul 24 17:24:23.348552 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.348145 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c679949f-tbzdn" Jul 24 17:24:23.348552 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.348191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c679949f-tbzdn" event={"ID":"2d3554b2-4e43-45b9-8c1d-413b98737390","Type":"ContainerDied","Data":"9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c"} Jul 24 17:24:23.348552 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.348244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c679949f-tbzdn" event={"ID":"2d3554b2-4e43-45b9-8c1d-413b98737390","Type":"ContainerDied","Data":"75e4d9be5855bcc8590dfd4c710e30902d9bf1e4975fc10b8ae3946c178fe427"} Jul 24 17:24:23.348552 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.348270 2574 scope.go:117] "RemoveContainer" containerID="9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c" Jul 24 17:24:23.356875 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.356853 2574 scope.go:117] "RemoveContainer" containerID="9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c" Jul 24 17:24:23.357120 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:24:23.357102 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c\": container with ID starting with 9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c not found: ID does not exist" containerID="9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c" Jul 24 17:24:23.357172 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.357129 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c"} err="failed to get container status \"9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c\": rpc error: code = NotFound desc = could not find container \"9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c\": container with ID starting with 9cb8281d464713daca01b65b264298897ee06c162e83c5196c25f78b5f71ad2c not found: ID does not exist" Jul 24 17:24:23.371757 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.371716 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86c679949f-tbzdn"] Jul 24 17:24:23.377993 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.377971 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86c679949f-tbzdn"] Jul 24 17:24:23.800593 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:23.800564 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d3554b2-4e43-45b9-8c1d-413b98737390" path="/var/lib/kubelet/pods/2d3554b2-4e43-45b9-8c1d-413b98737390/volumes" Jul 24 17:24:43.226532 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.226497 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p"] Jul 24 17:24:43.227045 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.226787 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d3554b2-4e43-45b9-8c1d-413b98737390" containerName="console" Jul 24 17:24:43.227045 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.226798 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d3554b2-4e43-45b9-8c1d-413b98737390" containerName="console" Jul 24 17:24:43.227045 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.226849 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d3554b2-4e43-45b9-8c1d-413b98737390" containerName="console" Jul 24 17:24:43.231561 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.231545 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.234443 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.234423 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vtkhp\"" Jul 24 17:24:43.234647 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.234634 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 17:24:43.235790 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.235764 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 17:24:43.238954 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.238493 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p"] Jul 24 17:24:43.345097 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.345060 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.345097 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.345101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.345312 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.345175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdx2\" (UniqueName: \"kubernetes.io/projected/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-kube-api-access-xjdx2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.445688 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.445652 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdx2\" (UniqueName: \"kubernetes.io/projected/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-kube-api-access-xjdx2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.445688 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.445694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.445961 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.445718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.446134 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.446101 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.446134 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.446126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.455361 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.455337 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdx2\" (UniqueName: \"kubernetes.io/projected/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-kube-api-access-xjdx2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.541641 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.541566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:43.673100 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:43.673077 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p"] Jul 24 17:24:43.674916 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:24:43.674887 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a94f7a_10d3_4f27_9008_89fcfbf0b045.slice/crio-0747d42bd6d2f9ef889b0de1d3d73bf4c2a12b0677bf8bd28150e207653e4a44 WatchSource:0}: Error finding container 0747d42bd6d2f9ef889b0de1d3d73bf4c2a12b0677bf8bd28150e207653e4a44: Status 404 returned error can't find the container with id 0747d42bd6d2f9ef889b0de1d3d73bf4c2a12b0677bf8bd28150e207653e4a44 Jul 24 17:24:44.427692 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:44.427602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" event={"ID":"b4a94f7a-10d3-4f27-9008-89fcfbf0b045","Type":"ContainerStarted","Data":"0747d42bd6d2f9ef889b0de1d3d73bf4c2a12b0677bf8bd28150e207653e4a44"} Jul 24 17:24:49.453695 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:49.453656 2574 generic.go:356] "Generic (PLEG): container finished" podID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerID="4a96eee483d6ec624e815180fc0c2962024636649a4d3350a37d3b36aa145385" exitCode=0 Jul 24 17:24:49.454125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:49.453712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" event={"ID":"b4a94f7a-10d3-4f27-9008-89fcfbf0b045","Type":"ContainerDied","Data":"4a96eee483d6ec624e815180fc0c2962024636649a4d3350a37d3b36aa145385"} Jul 24 17:24:51.464572 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:51.464534 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" event={"ID":"b4a94f7a-10d3-4f27-9008-89fcfbf0b045","Type":"ContainerStarted","Data":"664ed3c7f1d7c0c90e0d3dd803533ef879c8d0bbe563a4d98584876ed2637f51"} Jul 24 17:24:52.475104 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:52.475071 2574 generic.go:356] "Generic (PLEG): container finished" podID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerID="664ed3c7f1d7c0c90e0d3dd803533ef879c8d0bbe563a4d98584876ed2637f51" exitCode=0 Jul 24 17:24:52.475518 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:52.475113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" event={"ID":"b4a94f7a-10d3-4f27-9008-89fcfbf0b045","Type":"ContainerDied","Data":"664ed3c7f1d7c0c90e0d3dd803533ef879c8d0bbe563a4d98584876ed2637f51"} Jul 24 17:24:58.502530 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:58.502486 2574 generic.go:356] "Generic (PLEG): container finished" podID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerID="4b4fbf11ad0724b2b83a5abd10fc5accf7d2e94e77ab0bcc39ea584b53db9676" exitCode=0 Jul 24 17:24:58.502937 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:58.502567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" event={"ID":"b4a94f7a-10d3-4f27-9008-89fcfbf0b045","Type":"ContainerDied","Data":"4b4fbf11ad0724b2b83a5abd10fc5accf7d2e94e77ab0bcc39ea584b53db9676"} Jul 24 17:24:59.635436 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.635408 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:24:59.685378 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.685341 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-bundle\") pod \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " Jul 24 17:24:59.685546 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.685416 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-util\") pod \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " Jul 24 17:24:59.685546 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.685451 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjdx2\" (UniqueName: \"kubernetes.io/projected/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-kube-api-access-xjdx2\") pod \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\" (UID: \"b4a94f7a-10d3-4f27-9008-89fcfbf0b045\") " Jul 24 17:24:59.686133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.686101 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-bundle" (OuterVolumeSpecName: "bundle") pod "b4a94f7a-10d3-4f27-9008-89fcfbf0b045" (UID: "b4a94f7a-10d3-4f27-9008-89fcfbf0b045"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:24:59.687831 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.687804 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-kube-api-access-xjdx2" (OuterVolumeSpecName: "kube-api-access-xjdx2") pod "b4a94f7a-10d3-4f27-9008-89fcfbf0b045" (UID: "b4a94f7a-10d3-4f27-9008-89fcfbf0b045"). InnerVolumeSpecName "kube-api-access-xjdx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:24:59.690285 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.690266 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-util" (OuterVolumeSpecName: "util") pod "b4a94f7a-10d3-4f27-9008-89fcfbf0b045" (UID: "b4a94f7a-10d3-4f27-9008-89fcfbf0b045"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:24:59.786412 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.786317 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdx2\" (UniqueName: \"kubernetes.io/projected/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-kube-api-access-xjdx2\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:24:59.786412 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.786351 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-bundle\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:24:59.786412 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:24:59.786359 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4a94f7a-10d3-4f27-9008-89fcfbf0b045-util\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:25:00.512626 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:00.512591 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" Jul 24 17:25:00.512826 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:00.512591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ztl2p" event={"ID":"b4a94f7a-10d3-4f27-9008-89fcfbf0b045","Type":"ContainerDied","Data":"0747d42bd6d2f9ef889b0de1d3d73bf4c2a12b0677bf8bd28150e207653e4a44"} Jul 24 17:25:00.512826 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:00.512706 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0747d42bd6d2f9ef889b0de1d3d73bf4c2a12b0677bf8bd28150e207653e4a44" Jul 24 17:25:06.216480 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.216444 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl"] Jul 24 17:25:06.216901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.216757 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerName="pull" Jul 24 17:25:06.216901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.216768 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerName="pull" Jul 24 17:25:06.216901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.216788 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerName="extract" Jul 24 17:25:06.216901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.216794 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerName="extract" Jul 24 17:25:06.216901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.216807 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerName="util" Jul 24 17:25:06.216901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.216812 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerName="util" Jul 24 17:25:06.216901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.216865 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a94f7a-10d3-4f27-9008-89fcfbf0b045" containerName="extract" Jul 24 17:25:06.260946 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.260903 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl"] Jul 24 17:25:06.261105 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.261015 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" Jul 24 17:25:06.263894 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.263872 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-x8hs4\"" Jul 24 17:25:06.263894 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.263895 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:25:06.264098 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.263996 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 24 17:25:06.341225 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.341182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khjk9\" (UniqueName: \"kubernetes.io/projected/b4ce1a45-7034-44cf-8a3a-b63de78c6a59-kube-api-access-khjk9\") pod \"cert-manager-operator-controller-manager-55c444bf86-pq9dl\" (UID: \"b4ce1a45-7034-44cf-8a3a-b63de78c6a59\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" Jul 24 17:25:06.341399 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.341241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4ce1a45-7034-44cf-8a3a-b63de78c6a59-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pq9dl\" (UID: \"b4ce1a45-7034-44cf-8a3a-b63de78c6a59\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" Jul 24 17:25:06.442231 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.442194 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4ce1a45-7034-44cf-8a3a-b63de78c6a59-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pq9dl\" (UID: \"b4ce1a45-7034-44cf-8a3a-b63de78c6a59\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" Jul 24 17:25:06.442369 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.442290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjk9\" (UniqueName: \"kubernetes.io/projected/b4ce1a45-7034-44cf-8a3a-b63de78c6a59-kube-api-access-khjk9\") pod \"cert-manager-operator-controller-manager-55c444bf86-pq9dl\" (UID: \"b4ce1a45-7034-44cf-8a3a-b63de78c6a59\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" Jul 24 17:25:06.442588 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.442567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4ce1a45-7034-44cf-8a3a-b63de78c6a59-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pq9dl\" (UID: \"b4ce1a45-7034-44cf-8a3a-b63de78c6a59\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" Jul 24 17:25:06.451347 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.451318 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjk9\" (UniqueName: \"kubernetes.io/projected/b4ce1a45-7034-44cf-8a3a-b63de78c6a59-kube-api-access-khjk9\") pod \"cert-manager-operator-controller-manager-55c444bf86-pq9dl\" (UID: \"b4ce1a45-7034-44cf-8a3a-b63de78c6a59\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" Jul 24 17:25:06.575218 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.575133 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" Jul 24 17:25:06.732225 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:06.732192 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl"] Jul 24 17:25:06.735973 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:25:06.735946 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ce1a45_7034_44cf_8a3a_b63de78c6a59.slice/crio-78a4d61c25aadb1c7f02a4ba007e69578553b4f9e2a3c9dde286ecb743e56bfc WatchSource:0}: Error finding container 78a4d61c25aadb1c7f02a4ba007e69578553b4f9e2a3c9dde286ecb743e56bfc: Status 404 returned error can't find the container with id 78a4d61c25aadb1c7f02a4ba007e69578553b4f9e2a3c9dde286ecb743e56bfc Jul 24 17:25:07.540118 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:07.540084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" event={"ID":"b4ce1a45-7034-44cf-8a3a-b63de78c6a59","Type":"ContainerStarted","Data":"78a4d61c25aadb1c7f02a4ba007e69578553b4f9e2a3c9dde286ecb743e56bfc"} Jul 24 17:25:10.553824 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.553788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" event={"ID":"b4ce1a45-7034-44cf-8a3a-b63de78c6a59","Type":"ContainerStarted","Data":"1f39404f2cbc07d3ef831a1b471f5efc962e682b16e2944803aaaad41813ce59"} Jul 24 17:25:10.581650 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.581600 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pq9dl" podStartSLOduration=1.673518702 podStartE2EDuration="4.581583394s" totalImagesPullingTime="2.908064692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:25:10.579432049 +0000 UTC m=+467.245006687" watchObservedRunningTime="2026-07-24 17:25:10.581583394 +0000 UTC m=+467.247158030" Jul 24 17:25:10.592491 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.592463 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nrjzw"] Jul 24 17:25:10.613443 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.613422 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nrjzw"] Jul 24 17:25:10.613592 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.613508 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:10.617328 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.617309 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-knw4f\"" Jul 24 17:25:10.617437 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.617350 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 17:25:10.617909 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.617892 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 17:25:10.780650 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.780614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26544ee3-bcd9-4e50-8399-7ff0b8cb5f72-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nrjzw\" (UID: \"26544ee3-bcd9-4e50-8399-7ff0b8cb5f72\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:10.780856 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.780701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552rz\" (UniqueName: \"kubernetes.io/projected/26544ee3-bcd9-4e50-8399-7ff0b8cb5f72-kube-api-access-552rz\") pod \"cert-manager-webhook-6486ffd997-nrjzw\" (UID: \"26544ee3-bcd9-4e50-8399-7ff0b8cb5f72\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:10.881252 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.881161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552rz\" (UniqueName: \"kubernetes.io/projected/26544ee3-bcd9-4e50-8399-7ff0b8cb5f72-kube-api-access-552rz\") pod \"cert-manager-webhook-6486ffd997-nrjzw\" (UID: \"26544ee3-bcd9-4e50-8399-7ff0b8cb5f72\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:10.881252 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.881214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26544ee3-bcd9-4e50-8399-7ff0b8cb5f72-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nrjzw\" (UID: \"26544ee3-bcd9-4e50-8399-7ff0b8cb5f72\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:10.892909 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.892887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-552rz\" (UniqueName: \"kubernetes.io/projected/26544ee3-bcd9-4e50-8399-7ff0b8cb5f72-kube-api-access-552rz\") pod \"cert-manager-webhook-6486ffd997-nrjzw\" (UID: \"26544ee3-bcd9-4e50-8399-7ff0b8cb5f72\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:10.893110 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.893088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26544ee3-bcd9-4e50-8399-7ff0b8cb5f72-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nrjzw\" (UID: \"26544ee3-bcd9-4e50-8399-7ff0b8cb5f72\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:10.938042 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:10.938015 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:11.095241 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:11.095175 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nrjzw"] Jul 24 17:25:11.098701 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:25:11.098670 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26544ee3_bcd9_4e50_8399_7ff0b8cb5f72.slice/crio-a83a68fed360c4f5dfa13ef32c94c5649cc1ae1d825e1fe097b1969198c84008 WatchSource:0}: Error finding container a83a68fed360c4f5dfa13ef32c94c5649cc1ae1d825e1fe097b1969198c84008: Status 404 returned error can't find the container with id a83a68fed360c4f5dfa13ef32c94c5649cc1ae1d825e1fe097b1969198c84008 Jul 24 17:25:11.563560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:11.563527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" event={"ID":"26544ee3-bcd9-4e50-8399-7ff0b8cb5f72","Type":"ContainerStarted","Data":"a83a68fed360c4f5dfa13ef32c94c5649cc1ae1d825e1fe097b1969198c84008"} Jul 24 17:25:14.577629 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:14.577591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" event={"ID":"26544ee3-bcd9-4e50-8399-7ff0b8cb5f72","Type":"ContainerStarted","Data":"bdcabe83d9001e618fbd81b5360d19f13f3a08eb47ee2f9f404479e30936bfc1"} Jul 24 17:25:14.578042 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:14.577716 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:14.596542 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:14.596494 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" podStartSLOduration=2.077995551 podStartE2EDuration="4.596481823s" totalImagesPullingTime="2.518486272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:25:14.594257788 +0000 UTC m=+471.259832425" watchObservedRunningTime="2026-07-24 17:25:14.596481823 +0000 UTC m=+471.262056459" Jul 24 17:25:17.240118 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.240084 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn"] Jul 24 17:25:17.241961 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.241945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" Jul 24 17:25:17.245318 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.245298 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pjrg5\"" Jul 24 17:25:17.251960 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.251935 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn"] Jul 24 17:25:17.338078 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.338038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5382028c-a47d-42a7-9cbf-313a86c854ef-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jtnqn\" (UID: \"5382028c-a47d-42a7-9cbf-313a86c854ef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" Jul 24 17:25:17.338078 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.338075 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npcvc\" (UniqueName: \"kubernetes.io/projected/5382028c-a47d-42a7-9cbf-313a86c854ef-kube-api-access-npcvc\") pod \"cert-manager-cainjector-559cdd8c87-jtnqn\" (UID: \"5382028c-a47d-42a7-9cbf-313a86c854ef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" Jul 24 17:25:17.439467 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.439426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5382028c-a47d-42a7-9cbf-313a86c854ef-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jtnqn\" (UID: \"5382028c-a47d-42a7-9cbf-313a86c854ef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" Jul 24 17:25:17.439467 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.439468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npcvc\" (UniqueName: \"kubernetes.io/projected/5382028c-a47d-42a7-9cbf-313a86c854ef-kube-api-access-npcvc\") pod \"cert-manager-cainjector-559cdd8c87-jtnqn\" (UID: \"5382028c-a47d-42a7-9cbf-313a86c854ef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" Jul 24 17:25:17.448329 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.448301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npcvc\" (UniqueName: \"kubernetes.io/projected/5382028c-a47d-42a7-9cbf-313a86c854ef-kube-api-access-npcvc\") pod \"cert-manager-cainjector-559cdd8c87-jtnqn\" (UID: \"5382028c-a47d-42a7-9cbf-313a86c854ef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" Jul 24 17:25:17.448329 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.448313 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5382028c-a47d-42a7-9cbf-313a86c854ef-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jtnqn\" (UID: \"5382028c-a47d-42a7-9cbf-313a86c854ef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" Jul 24 17:25:17.551832 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.551752 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" Jul 24 17:25:17.678864 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:17.678834 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn"] Jul 24 17:25:17.681307 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:25:17.681282 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5382028c_a47d_42a7_9cbf_313a86c854ef.slice/crio-ca1cffd1b4888ce10bb5ec272264a9c4530237e6aeeb171b73dc61e53a75099c WatchSource:0}: Error finding container ca1cffd1b4888ce10bb5ec272264a9c4530237e6aeeb171b73dc61e53a75099c: Status 404 returned error can't find the container with id ca1cffd1b4888ce10bb5ec272264a9c4530237e6aeeb171b73dc61e53a75099c Jul 24 17:25:18.597356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:18.597326 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" event={"ID":"5382028c-a47d-42a7-9cbf-313a86c854ef","Type":"ContainerStarted","Data":"82448f3d147d6f5af90819f67427bf5720f36f06865ce19072751a1a67c249f9"} Jul 24 17:25:18.597356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:18.597361 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" event={"ID":"5382028c-a47d-42a7-9cbf-313a86c854ef","Type":"ContainerStarted","Data":"ca1cffd1b4888ce10bb5ec272264a9c4530237e6aeeb171b73dc61e53a75099c"} Jul 24 17:25:18.613610 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:18.613530 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtnqn" podStartSLOduration=1.613513099 podStartE2EDuration="1.613513099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:25:18.613164038 +0000 UTC m=+475.278738674" watchObservedRunningTime="2026-07-24 17:25:18.613513099 +0000 UTC m=+475.279087735" Jul 24 17:25:20.584009 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:20.583975 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nrjzw" Jul 24 17:25:24.085301 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.085257 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd"] Jul 24 17:25:24.087672 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.087649 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.091398 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.091380 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vtkhp\"" Jul 24 17:25:24.091501 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.091397 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 17:25:24.091501 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.091404 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 17:25:24.096611 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.096576 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd"] Jul 24 17:25:24.197583 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.197547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.197778 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.197622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.197778 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.197647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gjg5\" (UniqueName: \"kubernetes.io/projected/935e1f3b-ea49-40b3-b003-5b0b43fb6837-kube-api-access-5gjg5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.299057 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.299022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.299245 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.299087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.299245 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.299113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gjg5\" (UniqueName: \"kubernetes.io/projected/935e1f3b-ea49-40b3-b003-5b0b43fb6837-kube-api-access-5gjg5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.299429 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.299406 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.299483 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.299425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.308106 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.308087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gjg5\" (UniqueName: \"kubernetes.io/projected/935e1f3b-ea49-40b3-b003-5b0b43fb6837-kube-api-access-5gjg5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.397200 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.397132 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:24.526762 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.526674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd"] Jul 24 17:25:24.529061 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:25:24.529035 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935e1f3b_ea49_40b3_b003_5b0b43fb6837.slice/crio-708133fa7cee04e6ebf5bde4f476abbf79142d0fdfb0280f91b458851399a194 WatchSource:0}: Error finding container 708133fa7cee04e6ebf5bde4f476abbf79142d0fdfb0280f91b458851399a194: Status 404 returned error can't find the container with id 708133fa7cee04e6ebf5bde4f476abbf79142d0fdfb0280f91b458851399a194 Jul 24 17:25:24.631291 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.631259 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" event={"ID":"935e1f3b-ea49-40b3-b003-5b0b43fb6837","Type":"ContainerStarted","Data":"435dc0d04410004a326ba65f0b9fce41c25e7f1ba6955f882d5f0852ed6812f7"} Jul 24 17:25:24.631415 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:24.631319 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" event={"ID":"935e1f3b-ea49-40b3-b003-5b0b43fb6837","Type":"ContainerStarted","Data":"708133fa7cee04e6ebf5bde4f476abbf79142d0fdfb0280f91b458851399a194"} Jul 24 17:25:25.636740 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:25.636638 2574 generic.go:356] "Generic (PLEG): container finished" podID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerID="435dc0d04410004a326ba65f0b9fce41c25e7f1ba6955f882d5f0852ed6812f7" exitCode=0 Jul 24 17:25:25.636740 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:25.636689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" event={"ID":"935e1f3b-ea49-40b3-b003-5b0b43fb6837","Type":"ContainerDied","Data":"435dc0d04410004a326ba65f0b9fce41c25e7f1ba6955f882d5f0852ed6812f7"} Jul 24 17:25:27.647619 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:27.647586 2574 generic.go:356] "Generic (PLEG): container finished" podID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerID="7ae71c3939076385a564ceff8407be58c4cd028e3676efb3076288f63a12b4ab" exitCode=0 Jul 24 17:25:27.648057 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:27.647671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" event={"ID":"935e1f3b-ea49-40b3-b003-5b0b43fb6837","Type":"ContainerDied","Data":"7ae71c3939076385a564ceff8407be58c4cd028e3676efb3076288f63a12b4ab"} Jul 24 17:25:28.654405 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:28.654364 2574 generic.go:356] "Generic (PLEG): container finished" podID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerID="1fba390d5f49e0b9207a65444ec6d86dc23dc0f0779407df02b3bd52e474dc7d" exitCode=0 Jul 24 17:25:28.654809 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:28.654445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" event={"ID":"935e1f3b-ea49-40b3-b003-5b0b43fb6837","Type":"ContainerDied","Data":"1fba390d5f49e0b9207a65444ec6d86dc23dc0f0779407df02b3bd52e474dc7d"} Jul 24 17:25:29.784415 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.784389 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:29.839990 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.839964 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-util\") pod \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " Jul 24 17:25:29.840147 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.840000 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gjg5\" (UniqueName: \"kubernetes.io/projected/935e1f3b-ea49-40b3-b003-5b0b43fb6837-kube-api-access-5gjg5\") pod \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " Jul 24 17:25:29.840147 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.840061 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-bundle\") pod \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\" (UID: \"935e1f3b-ea49-40b3-b003-5b0b43fb6837\") " Jul 24 17:25:29.840486 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.840455 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-bundle" (OuterVolumeSpecName: "bundle") pod "935e1f3b-ea49-40b3-b003-5b0b43fb6837" (UID: "935e1f3b-ea49-40b3-b003-5b0b43fb6837"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:25:29.842335 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.842311 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935e1f3b-ea49-40b3-b003-5b0b43fb6837-kube-api-access-5gjg5" (OuterVolumeSpecName: "kube-api-access-5gjg5") pod "935e1f3b-ea49-40b3-b003-5b0b43fb6837" (UID: "935e1f3b-ea49-40b3-b003-5b0b43fb6837"). InnerVolumeSpecName "kube-api-access-5gjg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:25:29.844296 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.844270 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-util" (OuterVolumeSpecName: "util") pod "935e1f3b-ea49-40b3-b003-5b0b43fb6837" (UID: "935e1f3b-ea49-40b3-b003-5b0b43fb6837"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:25:29.940621 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.940544 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-util\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:25:29.940621 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.940569 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gjg5\" (UniqueName: \"kubernetes.io/projected/935e1f3b-ea49-40b3-b003-5b0b43fb6837-kube-api-access-5gjg5\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:25:29.940621 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:29.940581 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/935e1f3b-ea49-40b3-b003-5b0b43fb6837-bundle\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:25:30.665220 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:30.665197 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" Jul 24 17:25:30.665370 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:30.665199 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxdhmd" event={"ID":"935e1f3b-ea49-40b3-b003-5b0b43fb6837","Type":"ContainerDied","Data":"708133fa7cee04e6ebf5bde4f476abbf79142d0fdfb0280f91b458851399a194"} Jul 24 17:25:30.665370 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:30.665310 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="708133fa7cee04e6ebf5bde4f476abbf79142d0fdfb0280f91b458851399a194" Jul 24 17:25:31.373815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.373784 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fzsr8"] Jul 24 17:25:31.374221 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.374051 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerName="extract" Jul 24 17:25:31.374221 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.374061 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerName="extract" Jul 24 17:25:31.374221 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.374075 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerName="util" Jul 24 17:25:31.374221 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.374080 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerName="util" Jul 24 17:25:31.374221 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.374098 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerName="pull" Jul 24 17:25:31.374221 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.374103 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerName="pull" Jul 24 17:25:31.374221 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.374145 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="935e1f3b-ea49-40b3-b003-5b0b43fb6837" containerName="extract" Jul 24 17:25:31.375889 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.375870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fzsr8" Jul 24 17:25:31.378858 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.378836 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-j4q6b\"" Jul 24 17:25:31.384990 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.384970 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fzsr8"] Jul 24 17:25:31.451263 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.451234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwqmf\" (UniqueName: \"kubernetes.io/projected/8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d-kube-api-access-lwqmf\") pod \"cert-manager-55b4d954d5-fzsr8\" (UID: \"8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d\") " pod="cert-manager/cert-manager-55b4d954d5-fzsr8" Jul 24 17:25:31.451393 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.451306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d-bound-sa-token\") pod \"cert-manager-55b4d954d5-fzsr8\" (UID: \"8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d\") " pod="cert-manager/cert-manager-55b4d954d5-fzsr8" Jul 24 17:25:31.551934 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.551907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwqmf\" (UniqueName: \"kubernetes.io/projected/8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d-kube-api-access-lwqmf\") pod \"cert-manager-55b4d954d5-fzsr8\" (UID: \"8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d\") " pod="cert-manager/cert-manager-55b4d954d5-fzsr8" Jul 24 17:25:31.552054 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.551972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d-bound-sa-token\") pod \"cert-manager-55b4d954d5-fzsr8\" (UID: \"8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d\") " pod="cert-manager/cert-manager-55b4d954d5-fzsr8" Jul 24 17:25:31.560892 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.560865 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d-bound-sa-token\") pod \"cert-manager-55b4d954d5-fzsr8\" (UID: \"8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d\") " pod="cert-manager/cert-manager-55b4d954d5-fzsr8" Jul 24 17:25:31.560892 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.560897 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwqmf\" (UniqueName: \"kubernetes.io/projected/8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d-kube-api-access-lwqmf\") pod \"cert-manager-55b4d954d5-fzsr8\" (UID: \"8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d\") " pod="cert-manager/cert-manager-55b4d954d5-fzsr8" Jul 24 17:25:31.686927 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.686847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fzsr8" Jul 24 17:25:31.814374 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:31.814337 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fzsr8"] Jul 24 17:25:31.816545 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:25:31.816514 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c9b7b3f_9aba_40e5_9d6a_6f252f30f84d.slice/crio-8bba12c1fdccc71bbb1b37af821603d2de3c017553d5f32f660dfa02fe2e5872 WatchSource:0}: Error finding container 8bba12c1fdccc71bbb1b37af821603d2de3c017553d5f32f660dfa02fe2e5872: Status 404 returned error can't find the container with id 8bba12c1fdccc71bbb1b37af821603d2de3c017553d5f32f660dfa02fe2e5872 Jul 24 17:25:32.675044 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:32.675010 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fzsr8" event={"ID":"8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d","Type":"ContainerStarted","Data":"ae7acbb829a0b085f010002ff2799e866096de33fab981b2a278cd51367507b2"} Jul 24 17:25:32.675044 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:32.675047 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fzsr8" event={"ID":"8c9b7b3f-9aba-40e5-9d6a-6f252f30f84d","Type":"ContainerStarted","Data":"8bba12c1fdccc71bbb1b37af821603d2de3c017553d5f32f660dfa02fe2e5872"} Jul 24 17:25:32.693449 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:32.693400 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fzsr8" podStartSLOduration=1.693387275 podStartE2EDuration="1.693387275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:25:32.691666809 +0000 UTC m=+489.357241445" watchObservedRunningTime="2026-07-24 17:25:32.693387275 +0000 UTC m=+489.358961912" Jul 24 17:25:44.039367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.039331 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk"] Jul 24 17:25:44.043979 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.043960 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.049912 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.049887 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 17:25:44.051519 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.051502 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vtkhp\"" Jul 24 17:25:44.051649 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.051528 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 17:25:44.058677 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.058656 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk"] Jul 24 17:25:44.151868 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.151840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr6l9\" (UniqueName: \"kubernetes.io/projected/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-kube-api-access-hr6l9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.152016 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.151875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.152016 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.151898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.252422 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.252392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr6l9\" (UniqueName: \"kubernetes.io/projected/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-kube-api-access-hr6l9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.252563 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.252430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.252563 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.252454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.252793 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.252774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.252860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.252843 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.262150 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.262130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr6l9\" (UniqueName: \"kubernetes.io/projected/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-kube-api-access-hr6l9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.352964 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.352897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:44.484294 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.484274 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk"] Jul 24 17:25:44.487013 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:25:44.486985 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5dfcf4a_d4aa_4363_b285_d30d3919c05d.slice/crio-2ba59cb7724e5a51e0482cf34400d1db5b79f732a3e9a1880391f0402c844fc9 WatchSource:0}: Error finding container 2ba59cb7724e5a51e0482cf34400d1db5b79f732a3e9a1880391f0402c844fc9: Status 404 returned error can't find the container with id 2ba59cb7724e5a51e0482cf34400d1db5b79f732a3e9a1880391f0402c844fc9 Jul 24 17:25:44.724940 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.724907 2574 generic.go:356] "Generic (PLEG): container finished" podID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerID="66e2df542f08cb8437b72ceeba3755f4141cad42dffc0c7cff2ecb58f19f09e5" exitCode=0 Jul 24 17:25:44.725109 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.724986 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" event={"ID":"b5dfcf4a-d4aa-4363-b285-d30d3919c05d","Type":"ContainerDied","Data":"66e2df542f08cb8437b72ceeba3755f4141cad42dffc0c7cff2ecb58f19f09e5"} Jul 24 17:25:44.725109 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:44.725039 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" event={"ID":"b5dfcf4a-d4aa-4363-b285-d30d3919c05d","Type":"ContainerStarted","Data":"2ba59cb7724e5a51e0482cf34400d1db5b79f732a3e9a1880391f0402c844fc9"} Jul 24 17:25:45.731010 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:45.730980 2574 generic.go:356] "Generic (PLEG): container finished" podID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerID="94e37357bf3f629527b9fd0ea94550cc4b992fbe174ab6f91246db25194f1ae9" exitCode=0 Jul 24 17:25:45.731399 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:45.731015 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" event={"ID":"b5dfcf4a-d4aa-4363-b285-d30d3919c05d","Type":"ContainerDied","Data":"94e37357bf3f629527b9fd0ea94550cc4b992fbe174ab6f91246db25194f1ae9"} Jul 24 17:25:46.737048 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:46.737016 2574 generic.go:356] "Generic (PLEG): container finished" podID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerID="22bf63d9f5d193e0033c4d79aa98cb29765b3f9f172b7cd68f24d860d55b7cde" exitCode=0 Jul 24 17:25:46.737399 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:46.737095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" event={"ID":"b5dfcf4a-d4aa-4363-b285-d30d3919c05d","Type":"ContainerDied","Data":"22bf63d9f5d193e0033c4d79aa98cb29765b3f9f172b7cd68f24d860d55b7cde"} Jul 24 17:25:47.871172 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:47.871149 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:47.982821 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:47.982789 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-util\") pod \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " Jul 24 17:25:47.982996 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:47.982865 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr6l9\" (UniqueName: \"kubernetes.io/projected/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-kube-api-access-hr6l9\") pod \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " Jul 24 17:25:47.982996 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:47.982921 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-bundle\") pod \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\" (UID: \"b5dfcf4a-d4aa-4363-b285-d30d3919c05d\") " Jul 24 17:25:47.983678 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:47.983596 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-bundle" (OuterVolumeSpecName: "bundle") pod "b5dfcf4a-d4aa-4363-b285-d30d3919c05d" (UID: "b5dfcf4a-d4aa-4363-b285-d30d3919c05d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:25:47.985130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:47.985102 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-kube-api-access-hr6l9" (OuterVolumeSpecName: "kube-api-access-hr6l9") pod "b5dfcf4a-d4aa-4363-b285-d30d3919c05d" (UID: "b5dfcf4a-d4aa-4363-b285-d30d3919c05d"). InnerVolumeSpecName "kube-api-access-hr6l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:25:47.988517 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:47.988497 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-util" (OuterVolumeSpecName: "util") pod "b5dfcf4a-d4aa-4363-b285-d30d3919c05d" (UID: "b5dfcf4a-d4aa-4363-b285-d30d3919c05d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:25:48.084114 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:48.084020 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-bundle\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:25:48.084114 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:48.084054 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-util\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:25:48.084114 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:48.084063 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr6l9\" (UniqueName: \"kubernetes.io/projected/b5dfcf4a-d4aa-4363-b285-d30d3919c05d-kube-api-access-hr6l9\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:25:48.747168 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:48.747138 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" Jul 24 17:25:48.747327 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:48.747134 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af49bvk" event={"ID":"b5dfcf4a-d4aa-4363-b285-d30d3919c05d","Type":"ContainerDied","Data":"2ba59cb7724e5a51e0482cf34400d1db5b79f732a3e9a1880391f0402c844fc9"} Jul 24 17:25:48.747327 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:48.747284 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ba59cb7724e5a51e0482cf34400d1db5b79f732a3e9a1880391f0402c844fc9" Jul 24 17:25:56.108603 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.108569 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq"] Jul 24 17:25:56.109103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.108900 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerName="util" Jul 24 17:25:56.109103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.108911 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerName="util" Jul 24 17:25:56.109103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.108920 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerName="extract" Jul 24 17:25:56.109103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.108925 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerName="extract" Jul 24 17:25:56.109103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.108934 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerName="pull" Jul 24 17:25:56.109103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.108939 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerName="pull" Jul 24 17:25:56.109103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.109001 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5dfcf4a-d4aa-4363-b285-d30d3919c05d" containerName="extract" Jul 24 17:25:56.112202 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.112183 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.115477 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.115451 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 17:25:56.115583 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.115538 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 17:25:56.115583 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.115562 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 17:25:56.115685 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.115602 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 17:25:56.115885 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.115872 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xxn55\"" Jul 24 17:25:56.137961 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.137931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq"] Jul 24 17:25:56.254214 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.254180 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-webhook-cert\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.254379 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.254222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nb7\" (UniqueName: \"kubernetes.io/projected/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-kube-api-access-q6nb7\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.254379 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.254317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.254379 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.254353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-manifests\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.254484 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.254392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-tmp\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.355690 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.355650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.355690 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.355687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-manifests\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.355934 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.355710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-tmp\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.355934 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.355922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-webhook-cert\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.356029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.355976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nb7\" (UniqueName: \"kubernetes.io/projected/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-kube-api-access-q6nb7\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.356193 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.356163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-manifests\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.358292 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.358259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-tmp\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.358421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.358293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.358421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.358383 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-webhook-cert\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.364335 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.364278 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nb7\" (UniqueName: \"kubernetes.io/projected/6bbe55c5-07d5-4f94-a80b-15449eb8ca14-kube-api-access-q6nb7\") pod \"opendatahub-operator-controller-manager-8574785b6-phgvq\" (UID: \"6bbe55c5-07d5-4f94-a80b-15449eb8ca14\") " pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.422423 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.422396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:25:56.570102 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.570038 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq"] Jul 24 17:25:56.572387 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:25:56.572362 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbe55c5_07d5_4f94_a80b_15449eb8ca14.slice/crio-5b78ce2f472f1fb14ec17d9b8322b371d6fac3c3dbfa4794f1f174ea96627574 WatchSource:0}: Error finding container 5b78ce2f472f1fb14ec17d9b8322b371d6fac3c3dbfa4794f1f174ea96627574: Status 404 returned error can't find the container with id 5b78ce2f472f1fb14ec17d9b8322b371d6fac3c3dbfa4794f1f174ea96627574 Jul 24 17:25:56.782318 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:56.782277 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" event={"ID":"6bbe55c5-07d5-4f94-a80b-15449eb8ca14","Type":"ContainerStarted","Data":"5b78ce2f472f1fb14ec17d9b8322b371d6fac3c3dbfa4794f1f174ea96627574"} Jul 24 17:25:59.801834 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:25:59.801763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" event={"ID":"6bbe55c5-07d5-4f94-a80b-15449eb8ca14","Type":"ContainerStarted","Data":"4837f97c3bb6cd988acc4cf8d061c7a6eea3e8cad744764c1950c2663f122435"} Jul 24 17:26:00.808110 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:00.808033 2574 generic.go:356] "Generic (PLEG): container finished" podID="6bbe55c5-07d5-4f94-a80b-15449eb8ca14" containerID="4837f97c3bb6cd988acc4cf8d061c7a6eea3e8cad744764c1950c2663f122435" exitCode=0 Jul 24 17:26:00.808110 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:00.808068 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" event={"ID":"6bbe55c5-07d5-4f94-a80b-15449eb8ca14","Type":"ContainerDied","Data":"4837f97c3bb6cd988acc4cf8d061c7a6eea3e8cad744764c1950c2663f122435"} Jul 24 17:26:01.814637 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:01.814592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" event={"ID":"6bbe55c5-07d5-4f94-a80b-15449eb8ca14","Type":"ContainerStarted","Data":"c1b861c41347fbac9611eee43add116343b8904b52916c9995058d00268ec088"} Jul 24 17:26:01.815041 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:01.814761 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:26:01.839184 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:01.839124 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" podStartSLOduration=1.498310862 podStartE2EDuration="5.839102203s" totalImagesPullingTime="3.334586896s" totalInitContainerRuntime="1.006204445s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:25:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:01.836472365 +0000 UTC m=+518.502047012" watchObservedRunningTime="2026-07-24 17:26:01.839102203 +0000 UTC m=+518.504676842" Jul 24 17:26:12.823030 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:12.823002 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8574785b6-phgvq" Jul 24 17:26:16.206778 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.206674 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w"] Jul 24 17:26:16.220630 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.220605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.225301 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.225256 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 17:26:16.226179 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.226152 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 17:26:16.230473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.230456 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gmfwn\"" Jul 24 17:26:16.233650 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.233628 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w"] Jul 24 17:26:16.318716 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.318685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c3553b-4df1-4ccb-b888-8157824c6fa2-tls-certs\") pod \"kube-auth-proxy-956b69cf9-5vh8w\" (UID: \"c4c3553b-4df1-4ccb-b888-8157824c6fa2\") " pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.318874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.318744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4c3553b-4df1-4ccb-b888-8157824c6fa2-tmp\") pod \"kube-auth-proxy-956b69cf9-5vh8w\" (UID: \"c4c3553b-4df1-4ccb-b888-8157824c6fa2\") " pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.318874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.318820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gj7\" (UniqueName: \"kubernetes.io/projected/c4c3553b-4df1-4ccb-b888-8157824c6fa2-kube-api-access-c8gj7\") pod \"kube-auth-proxy-956b69cf9-5vh8w\" (UID: \"c4c3553b-4df1-4ccb-b888-8157824c6fa2\") " pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.420148 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.420111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gj7\" (UniqueName: \"kubernetes.io/projected/c4c3553b-4df1-4ccb-b888-8157824c6fa2-kube-api-access-c8gj7\") pod \"kube-auth-proxy-956b69cf9-5vh8w\" (UID: \"c4c3553b-4df1-4ccb-b888-8157824c6fa2\") " pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.420329 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.420219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c3553b-4df1-4ccb-b888-8157824c6fa2-tls-certs\") pod \"kube-auth-proxy-956b69cf9-5vh8w\" (UID: \"c4c3553b-4df1-4ccb-b888-8157824c6fa2\") " pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.420329 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.420266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4c3553b-4df1-4ccb-b888-8157824c6fa2-tmp\") pod \"kube-auth-proxy-956b69cf9-5vh8w\" (UID: \"c4c3553b-4df1-4ccb-b888-8157824c6fa2\") " pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.422676 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.422647 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4c3553b-4df1-4ccb-b888-8157824c6fa2-tmp\") pod \"kube-auth-proxy-956b69cf9-5vh8w\" (UID: \"c4c3553b-4df1-4ccb-b888-8157824c6fa2\") " pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.423293 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.423267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c3553b-4df1-4ccb-b888-8157824c6fa2-tls-certs\") pod \"kube-auth-proxy-956b69cf9-5vh8w\" (UID: \"c4c3553b-4df1-4ccb-b888-8157824c6fa2\") " pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.451323 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.451296 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gj7\" (UniqueName: \"kubernetes.io/projected/c4c3553b-4df1-4ccb-b888-8157824c6fa2-kube-api-access-c8gj7\") pod \"kube-auth-proxy-956b69cf9-5vh8w\" (UID: \"c4c3553b-4df1-4ccb-b888-8157824c6fa2\") " pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.531358 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.531321 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" Jul 24 17:26:16.687421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.687389 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w"] Jul 24 17:26:16.690591 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:26:16.690566 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c3553b_4df1_4ccb_b888_8157824c6fa2.slice/crio-c20325073ae13655f0e696e9d6aeace03ec213b7c51c98a8b6c7195c53ada5e8 WatchSource:0}: Error finding container c20325073ae13655f0e696e9d6aeace03ec213b7c51c98a8b6c7195c53ada5e8: Status 404 returned error can't find the container with id c20325073ae13655f0e696e9d6aeace03ec213b7c51c98a8b6c7195c53ada5e8 Jul 24 17:26:16.890797 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:16.890695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" event={"ID":"c4c3553b-4df1-4ccb-b888-8157824c6fa2","Type":"ContainerStarted","Data":"c20325073ae13655f0e696e9d6aeace03ec213b7c51c98a8b6c7195c53ada5e8"} Jul 24 17:26:18.090203 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.090169 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w"] Jul 24 17:26:18.093701 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.093683 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.097231 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.097207 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 17:26:18.097375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.097211 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 17:26:18.097375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.097276 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ncljf\"" Jul 24 17:26:18.097375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.097320 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 17:26:18.097375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.097339 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 17:26:18.107531 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.107475 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w"] Jul 24 17:26:18.236774 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.236716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.236960 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.236845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.236960 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.236882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.236960 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.236910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9504f785-e2cf-488e-9c19-5e77b06b7548-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.237125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.236959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.237125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.236995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.237125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.237027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnd6l\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-kube-api-access-bnd6l\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.337896 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.337857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.338080 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.337904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.338080 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.337930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9504f785-e2cf-488e-9c19-5e77b06b7548-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.338797 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.338362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.338797 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.338423 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.338797 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.338455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnd6l\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-kube-api-access-bnd6l\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.338797 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.338561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.338797 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.338578 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.341354 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.341292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.341590 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.341568 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9504f785-e2cf-488e-9c19-5e77b06b7548-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.342221 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.342201 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.343240 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.343216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.347670 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.347646 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnd6l\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-kube-api-access-bnd6l\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.348022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.347998 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7mn9w\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.406457 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.406426 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:18.958995 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:18.958972 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w"] Jul 24 17:26:18.960965 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:26:18.960941 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9504f785_e2cf_488e_9c19_5e77b06b7548.slice/crio-8bcfd2ef38f2f76d0c1b1024ef4d8ac58d176786d1ae5b84064c4523848d67b2 WatchSource:0}: Error finding container 8bcfd2ef38f2f76d0c1b1024ef4d8ac58d176786d1ae5b84064c4523848d67b2: Status 404 returned error can't find the container with id 8bcfd2ef38f2f76d0c1b1024ef4d8ac58d176786d1ae5b84064c4523848d67b2 Jul 24 17:26:19.913504 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:19.913463 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" event={"ID":"9504f785-e2cf-488e-9c19-5e77b06b7548","Type":"ContainerStarted","Data":"8bcfd2ef38f2f76d0c1b1024ef4d8ac58d176786d1ae5b84064c4523848d67b2"} Jul 24 17:26:19.916917 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:19.916879 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" event={"ID":"c4c3553b-4df1-4ccb-b888-8157824c6fa2","Type":"ContainerStarted","Data":"13c28e35778615af902b4a8fa689a14e616c681352cc6fc7909a3c2430150c65"} Jul 24 17:26:19.938122 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:19.938060 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-956b69cf9-5vh8w" podStartSLOduration=1.765269346 podStartE2EDuration="3.938039625s" totalImagesPullingTime="2.172770279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:19.936034316 +0000 UTC m=+536.601608964" watchObservedRunningTime="2026-07-24 17:26:19.938039625 +0000 UTC m=+536.603614263" Jul 24 17:26:21.647014 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:21.646963 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:26:21.647360 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:21.647056 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:26:21.929348 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:21.929266 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" event={"ID":"9504f785-e2cf-488e-9c19-5e77b06b7548","Type":"ContainerStarted","Data":"93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e"} Jul 24 17:26:21.929483 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:21.929363 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:21.954110 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:21.954057 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" podStartSLOduration=1.271028058 podStartE2EDuration="3.954038099s" totalImagesPullingTime="2.683010041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:21.952069755 +0000 UTC m=+538.617644404" watchObservedRunningTime="2026-07-24 17:26:21.954038099 +0000 UTC m=+538.619612740" Jul 24 17:26:22.935792 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:22.935765 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:27.669672 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.669636 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r"] Jul 24 17:26:27.674448 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.674427 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:27.677711 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.677692 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 17:26:27.679114 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.679096 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-p8rs9\"" Jul 24 17:26:27.681646 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.681620 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8"] Jul 24 17:26:27.685186 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.685056 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:27.686524 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.686504 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r"] Jul 24 17:26:27.688301 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.688276 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 17:26:27.691117 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.690938 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-l5k8m\"" Jul 24 17:26:27.703930 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.703909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8"] Jul 24 17:26:27.727234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.727207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2pt6\" (UniqueName: \"kubernetes.io/projected/84aabd4e-2a07-48e9-a3d5-1591954a1408-kube-api-access-f2pt6\") pod \"kserve-controller-manager-d6f4b65b8-kbd5r\" (UID: \"84aabd4e-2a07-48e9-a3d5-1591954a1408\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:27.727378 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.727329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84aabd4e-2a07-48e9-a3d5-1591954a1408-cert\") pod \"kserve-controller-manager-d6f4b65b8-kbd5r\" (UID: \"84aabd4e-2a07-48e9-a3d5-1591954a1408\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:27.828260 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.828228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5562ae78-dee0-4b9b-a693-626e0863dbf0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7s2g8\" (UID: \"5562ae78-dee0-4b9b-a693-626e0863dbf0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:27.828457 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.828264 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc47f\" (UniqueName: \"kubernetes.io/projected/5562ae78-dee0-4b9b-a693-626e0863dbf0-kube-api-access-dc47f\") pod \"llmisvc-controller-manager-777c6bd7d5-7s2g8\" (UID: \"5562ae78-dee0-4b9b-a693-626e0863dbf0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:27.828457 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.828305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2pt6\" (UniqueName: \"kubernetes.io/projected/84aabd4e-2a07-48e9-a3d5-1591954a1408-kube-api-access-f2pt6\") pod \"kserve-controller-manager-d6f4b65b8-kbd5r\" (UID: \"84aabd4e-2a07-48e9-a3d5-1591954a1408\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:27.828457 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.828406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84aabd4e-2a07-48e9-a3d5-1591954a1408-cert\") pod \"kserve-controller-manager-d6f4b65b8-kbd5r\" (UID: \"84aabd4e-2a07-48e9-a3d5-1591954a1408\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:27.828619 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:26:27.828524 2574 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 24 17:26:27.828619 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:26:27.828589 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84aabd4e-2a07-48e9-a3d5-1591954a1408-cert podName:84aabd4e-2a07-48e9-a3d5-1591954a1408 nodeName:}" failed. No retries permitted until 2026-07-24 17:26:28.328567641 +0000 UTC m=+544.994142255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/84aabd4e-2a07-48e9-a3d5-1591954a1408-cert") pod "kserve-controller-manager-d6f4b65b8-kbd5r" (UID: "84aabd4e-2a07-48e9-a3d5-1591954a1408") : secret "kserve-webhook-server-cert" not found Jul 24 17:26:27.844169 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.844140 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2pt6\" (UniqueName: \"kubernetes.io/projected/84aabd4e-2a07-48e9-a3d5-1591954a1408-kube-api-access-f2pt6\") pod \"kserve-controller-manager-d6f4b65b8-kbd5r\" (UID: \"84aabd4e-2a07-48e9-a3d5-1591954a1408\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:27.929083 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.928997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5562ae78-dee0-4b9b-a693-626e0863dbf0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7s2g8\" (UID: \"5562ae78-dee0-4b9b-a693-626e0863dbf0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:27.929083 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.929041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc47f\" (UniqueName: \"kubernetes.io/projected/5562ae78-dee0-4b9b-a693-626e0863dbf0-kube-api-access-dc47f\") pod \"llmisvc-controller-manager-777c6bd7d5-7s2g8\" (UID: \"5562ae78-dee0-4b9b-a693-626e0863dbf0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:27.929306 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:26:27.929154 2574 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 24 17:26:27.929306 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:26:27.929211 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5562ae78-dee0-4b9b-a693-626e0863dbf0-cert podName:5562ae78-dee0-4b9b-a693-626e0863dbf0 nodeName:}" failed. No retries permitted until 2026-07-24 17:26:28.429196365 +0000 UTC m=+545.094770980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5562ae78-dee0-4b9b-a693-626e0863dbf0-cert") pod "llmisvc-controller-manager-777c6bd7d5-7s2g8" (UID: "5562ae78-dee0-4b9b-a693-626e0863dbf0") : secret "llmisvc-webhook-server-cert" not found Jul 24 17:26:27.945694 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:27.945664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc47f\" (UniqueName: \"kubernetes.io/projected/5562ae78-dee0-4b9b-a693-626e0863dbf0-kube-api-access-dc47f\") pod \"llmisvc-controller-manager-777c6bd7d5-7s2g8\" (UID: \"5562ae78-dee0-4b9b-a693-626e0863dbf0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:28.330909 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.330874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84aabd4e-2a07-48e9-a3d5-1591954a1408-cert\") pod \"kserve-controller-manager-d6f4b65b8-kbd5r\" (UID: \"84aabd4e-2a07-48e9-a3d5-1591954a1408\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:28.333339 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.333322 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84aabd4e-2a07-48e9-a3d5-1591954a1408-cert\") pod \"kserve-controller-manager-d6f4b65b8-kbd5r\" (UID: \"84aabd4e-2a07-48e9-a3d5-1591954a1408\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:28.431789 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.431751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5562ae78-dee0-4b9b-a693-626e0863dbf0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7s2g8\" (UID: \"5562ae78-dee0-4b9b-a693-626e0863dbf0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:28.434401 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.434381 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5562ae78-dee0-4b9b-a693-626e0863dbf0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7s2g8\" (UID: \"5562ae78-dee0-4b9b-a693-626e0863dbf0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:28.587319 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.587242 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:28.598039 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.598010 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:28.737080 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.737056 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r"] Jul 24 17:26:28.739336 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:26:28.739312 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aabd4e_2a07_48e9_a3d5_1591954a1408.slice/crio-394d977860798310bdba1d4826a991aca3f3bb55b7750ba6eb97a45d778f277d WatchSource:0}: Error finding container 394d977860798310bdba1d4826a991aca3f3bb55b7750ba6eb97a45d778f277d: Status 404 returned error can't find the container with id 394d977860798310bdba1d4826a991aca3f3bb55b7750ba6eb97a45d778f277d Jul 24 17:26:28.756025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.755969 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8"] Jul 24 17:26:28.757943 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:26:28.757921 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5562ae78_dee0_4b9b_a693_626e0863dbf0.slice/crio-070249712b83a5eb0308fa13aa47ecf918b6638c32088000faa69eda12013dfa WatchSource:0}: Error finding container 070249712b83a5eb0308fa13aa47ecf918b6638c32088000faa69eda12013dfa: Status 404 returned error can't find the container with id 070249712b83a5eb0308fa13aa47ecf918b6638c32088000faa69eda12013dfa Jul 24 17:26:28.962273 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.962237 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" event={"ID":"84aabd4e-2a07-48e9-a3d5-1591954a1408","Type":"ContainerStarted","Data":"394d977860798310bdba1d4826a991aca3f3bb55b7750ba6eb97a45d778f277d"} Jul 24 17:26:28.963433 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:28.963413 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" event={"ID":"5562ae78-dee0-4b9b-a693-626e0863dbf0","Type":"ContainerStarted","Data":"070249712b83a5eb0308fa13aa47ecf918b6638c32088000faa69eda12013dfa"} Jul 24 17:26:30.979335 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:30.979304 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w"] Jul 24 17:26:30.979820 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:30.979506 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" podUID="9504f785-e2cf-488e-9c19-5e77b06b7548" containerName="discovery" containerID="cri-o://93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e" gracePeriod=30 Jul 24 17:26:31.831801 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.831780 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:31.862750 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.862695 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-cacerts\") pod \"9504f785-e2cf-488e-9c19-5e77b06b7548\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " Jul 24 17:26:31.862874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.862755 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9504f785-e2cf-488e-9c19-5e77b06b7548-local-certs\") pod \"9504f785-e2cf-488e-9c19-5e77b06b7548\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " Jul 24 17:26:31.862874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.862859 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-kubeconfig\") pod \"9504f785-e2cf-488e-9c19-5e77b06b7548\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " Jul 24 17:26:31.862965 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.862889 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-dns-cert\") pod \"9504f785-e2cf-488e-9c19-5e77b06b7548\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " Jul 24 17:26:31.862965 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.862917 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-ca-configmap\") pod \"9504f785-e2cf-488e-9c19-5e77b06b7548\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " Jul 24 17:26:31.863678 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.863148 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-token\") pod \"9504f785-e2cf-488e-9c19-5e77b06b7548\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " Jul 24 17:26:31.863678 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.863196 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnd6l\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-kube-api-access-bnd6l\") pod \"9504f785-e2cf-488e-9c19-5e77b06b7548\" (UID: \"9504f785-e2cf-488e-9c19-5e77b06b7548\") " Jul 24 17:26:31.863678 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.863236 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9504f785-e2cf-488e-9c19-5e77b06b7548" (UID: "9504f785-e2cf-488e-9c19-5e77b06b7548"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:26:31.863678 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.863672 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-ca-configmap\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:26:31.865650 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.865625 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-cacerts" (OuterVolumeSpecName: "cacerts") pod "9504f785-e2cf-488e-9c19-5e77b06b7548" (UID: "9504f785-e2cf-488e-9c19-5e77b06b7548"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:26:31.865810 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.865646 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9504f785-e2cf-488e-9c19-5e77b06b7548" (UID: "9504f785-e2cf-488e-9c19-5e77b06b7548"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:26:31.865917 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.865884 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9504f785-e2cf-488e-9c19-5e77b06b7548" (UID: "9504f785-e2cf-488e-9c19-5e77b06b7548"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:26:31.865996 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.865974 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9504f785-e2cf-488e-9c19-5e77b06b7548-local-certs" (OuterVolumeSpecName: "local-certs") pod "9504f785-e2cf-488e-9c19-5e77b06b7548" (UID: "9504f785-e2cf-488e-9c19-5e77b06b7548"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:26:31.866154 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.866132 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-token" (OuterVolumeSpecName: "istio-token") pod "9504f785-e2cf-488e-9c19-5e77b06b7548" (UID: "9504f785-e2cf-488e-9c19-5e77b06b7548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:26:31.866376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.866352 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-kube-api-access-bnd6l" (OuterVolumeSpecName: "kube-api-access-bnd6l") pod "9504f785-e2cf-488e-9c19-5e77b06b7548" (UID: "9504f785-e2cf-488e-9c19-5e77b06b7548"). InnerVolumeSpecName "kube-api-access-bnd6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:26:31.964529 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.964493 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-cacerts\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:26:31.964529 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.964531 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9504f785-e2cf-488e-9c19-5e77b06b7548-local-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:26:31.964742 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.964545 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-kubeconfig\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:26:31.964742 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.964553 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-csr-dns-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:26:31.964742 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.964562 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:26:31.964742 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.964571 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnd6l\" (UniqueName: \"kubernetes.io/projected/9504f785-e2cf-488e-9c19-5e77b06b7548-kube-api-access-bnd6l\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:26:31.983012 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.982979 2574 generic.go:356] "Generic (PLEG): container finished" podID="9504f785-e2cf-488e-9c19-5e77b06b7548" containerID="93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e" exitCode=0 Jul 24 17:26:31.983436 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.983031 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" Jul 24 17:26:31.983436 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.983068 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" event={"ID":"9504f785-e2cf-488e-9c19-5e77b06b7548","Type":"ContainerDied","Data":"93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e"} Jul 24 17:26:31.983436 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.983114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w" event={"ID":"9504f785-e2cf-488e-9c19-5e77b06b7548","Type":"ContainerDied","Data":"8bcfd2ef38f2f76d0c1b1024ef4d8ac58d176786d1ae5b84064c4523848d67b2"} Jul 24 17:26:31.983436 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.983140 2574 scope.go:117] "RemoveContainer" containerID="93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e" Jul 24 17:26:31.985275 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.985243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" event={"ID":"84aabd4e-2a07-48e9-a3d5-1591954a1408","Type":"ContainerStarted","Data":"9dfd8f077ff5704269cb7ec78ed4f071609b91089cd8108216a70ec8910ade97"} Jul 24 17:26:31.985490 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.985476 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:26:31.987152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.987130 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" event={"ID":"5562ae78-dee0-4b9b-a693-626e0863dbf0","Type":"ContainerStarted","Data":"5ba03bbed29d3ea660ef034cd9aad1874ca25d259016d1ed11112f4ca0d890a7"} Jul 24 17:26:31.987273 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.987259 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:26:31.993012 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.992996 2574 scope.go:117] "RemoveContainer" containerID="93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e" Jul 24 17:26:31.993300 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:26:31.993277 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e\": container with ID starting with 93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e not found: ID does not exist" containerID="93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e" Jul 24 17:26:31.993379 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:31.993306 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e"} err="failed to get container status \"93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e\": rpc error: code = NotFound desc = could not find container \"93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e\": container with ID starting with 93a9f10f8579b96922cf2adbafc6e216bb553a65211564a74506dd04ffde799e not found: ID does not exist" Jul 24 17:26:32.012376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:32.012332 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" podStartSLOduration=2.029292106 podStartE2EDuration="5.012319949s" totalImagesPullingTime="2.983027843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:32.010582715 +0000 UTC m=+548.676157365" watchObservedRunningTime="2026-07-24 17:26:32.012319949 +0000 UTC m=+548.677894585" Jul 24 17:26:32.034652 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:32.034604 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" podStartSLOduration=2.067248263 podStartE2EDuration="5.034591114s" totalImagesPullingTime="2.967342851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:32.033518328 +0000 UTC m=+548.699092966" watchObservedRunningTime="2026-07-24 17:26:32.034591114 +0000 UTC m=+548.700165751" Jul 24 17:26:32.050511 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:32.050484 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w"] Jul 24 17:26:32.054996 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:32.054968 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7mn9w"] Jul 24 17:26:33.801862 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:26:33.801829 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9504f785-e2cf-488e-9c19-5e77b06b7548" path="/var/lib/kubelet/pods/9504f785-e2cf-488e-9c19-5e77b06b7548/volumes" Jul 24 17:27:02.995627 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:02.995598 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7s2g8" Jul 24 17:27:02.998600 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:02.998581 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kbd5r" Jul 24 17:27:17.190829 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.190795 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qwd4f"] Jul 24 17:27:17.191223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.191112 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9504f785-e2cf-488e-9c19-5e77b06b7548" containerName="discovery" Jul 24 17:27:17.191223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.191123 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9504f785-e2cf-488e-9c19-5e77b06b7548" containerName="discovery" Jul 24 17:27:17.191223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.191171 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9504f785-e2cf-488e-9c19-5e77b06b7548" containerName="discovery" Jul 24 17:27:17.193064 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.193044 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" Jul 24 17:27:17.197961 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.197828 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-d9jj4\"" Jul 24 17:27:17.197961 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.197956 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 17:27:17.198164 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.198091 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 17:27:17.204496 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.204476 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qwd4f"] Jul 24 17:27:17.352185 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.352147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svxxv\" (UniqueName: \"kubernetes.io/projected/03e58c51-820b-4fae-9bbd-7f72624b2732-kube-api-access-svxxv\") pod \"kuadrant-operator-catalog-qwd4f\" (UID: \"03e58c51-820b-4fae-9bbd-7f72624b2732\") " pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" Jul 24 17:27:17.453592 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.453522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svxxv\" (UniqueName: \"kubernetes.io/projected/03e58c51-820b-4fae-9bbd-7f72624b2732-kube-api-access-svxxv\") pod \"kuadrant-operator-catalog-qwd4f\" (UID: \"03e58c51-820b-4fae-9bbd-7f72624b2732\") " pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" Jul 24 17:27:17.463495 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.463464 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svxxv\" (UniqueName: \"kubernetes.io/projected/03e58c51-820b-4fae-9bbd-7f72624b2732-kube-api-access-svxxv\") pod \"kuadrant-operator-catalog-qwd4f\" (UID: \"03e58c51-820b-4fae-9bbd-7f72624b2732\") " pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" Jul 24 17:27:17.504988 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.504960 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" Jul 24 17:27:17.558541 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.558512 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qwd4f"] Jul 24 17:27:17.633590 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.633566 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qwd4f"] Jul 24 17:27:17.636360 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:17.636325 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e58c51_820b_4fae_9bbd_7f72624b2732.slice/crio-8b210395b70dc4eebbe60f2087cc0fc9982845d14753b5d1d6ef6128f12f87e2 WatchSource:0}: Error finding container 8b210395b70dc4eebbe60f2087cc0fc9982845d14753b5d1d6ef6128f12f87e2: Status 404 returned error can't find the container with id 8b210395b70dc4eebbe60f2087cc0fc9982845d14753b5d1d6ef6128f12f87e2 Jul 24 17:27:17.776606 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.776578 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8txjb"] Jul 24 17:27:17.779264 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.779238 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8txjb" Jul 24 17:27:17.787639 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.787619 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8txjb"] Jul 24 17:27:17.957678 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:17.957644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqz2\" (UniqueName: \"kubernetes.io/projected/63d947fe-7ba0-49ce-b21f-22357ee10c26-kube-api-access-7xqz2\") pod \"kuadrant-operator-catalog-8txjb\" (UID: \"63d947fe-7ba0-49ce-b21f-22357ee10c26\") " pod="kuadrant-system/kuadrant-operator-catalog-8txjb" Jul 24 17:27:18.058438 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:18.058365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqz2\" (UniqueName: \"kubernetes.io/projected/63d947fe-7ba0-49ce-b21f-22357ee10c26-kube-api-access-7xqz2\") pod \"kuadrant-operator-catalog-8txjb\" (UID: \"63d947fe-7ba0-49ce-b21f-22357ee10c26\") " pod="kuadrant-system/kuadrant-operator-catalog-8txjb" Jul 24 17:27:18.068986 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:18.068954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqz2\" (UniqueName: \"kubernetes.io/projected/63d947fe-7ba0-49ce-b21f-22357ee10c26-kube-api-access-7xqz2\") pod \"kuadrant-operator-catalog-8txjb\" (UID: \"63d947fe-7ba0-49ce-b21f-22357ee10c26\") " pod="kuadrant-system/kuadrant-operator-catalog-8txjb" Jul 24 17:27:18.089943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:18.089913 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8txjb" Jul 24 17:27:18.206884 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:18.206843 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" event={"ID":"03e58c51-820b-4fae-9bbd-7f72624b2732","Type":"ContainerStarted","Data":"8b210395b70dc4eebbe60f2087cc0fc9982845d14753b5d1d6ef6128f12f87e2"} Jul 24 17:27:18.229963 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:18.229936 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8txjb"] Jul 24 17:27:18.316179 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:18.316084 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d947fe_7ba0_49ce_b21f_22357ee10c26.slice/crio-43cc13a2ec4f6664cb8aa407f7b8799928c507c8bccd6ee5642f33dc998619f2 WatchSource:0}: Error finding container 43cc13a2ec4f6664cb8aa407f7b8799928c507c8bccd6ee5642f33dc998619f2: Status 404 returned error can't find the container with id 43cc13a2ec4f6664cb8aa407f7b8799928c507c8bccd6ee5642f33dc998619f2 Jul 24 17:27:19.213903 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:19.213864 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8txjb" event={"ID":"63d947fe-7ba0-49ce-b21f-22357ee10c26","Type":"ContainerStarted","Data":"43cc13a2ec4f6664cb8aa407f7b8799928c507c8bccd6ee5642f33dc998619f2"} Jul 24 17:27:20.221264 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:20.221226 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" event={"ID":"03e58c51-820b-4fae-9bbd-7f72624b2732","Type":"ContainerStarted","Data":"1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4"} Jul 24 17:27:20.221742 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:20.221274 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" podUID="03e58c51-820b-4fae-9bbd-7f72624b2732" containerName="registry-server" containerID="cri-o://1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4" gracePeriod=2 Jul 24 17:27:20.223085 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:20.223018 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8txjb" event={"ID":"63d947fe-7ba0-49ce-b21f-22357ee10c26","Type":"ContainerStarted","Data":"d2c676be5bb542b11c7c42f03423eee5847768d46e172a31c7daf97fe36cdd32"} Jul 24 17:27:20.238874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:20.238835 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" podStartSLOduration=1.380157997 podStartE2EDuration="3.238822467s" totalImagesPullingTime="1.85866447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:27:20.23704573 +0000 UTC m=+596.902620367" watchObservedRunningTime="2026-07-24 17:27:20.238822467 +0000 UTC m=+596.904397105" Jul 24 17:27:20.253869 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:20.253830 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8txjb" podStartSLOduration=2.076162709 podStartE2EDuration="3.253818479s" totalImagesPullingTime="1.17765577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:27:20.252415711 +0000 UTC m=+596.917990362" watchObservedRunningTime="2026-07-24 17:27:20.253818479 +0000 UTC m=+596.919393119" Jul 24 17:27:20.475258 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:20.475206 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" Jul 24 17:27:20.479380 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:20.479361 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svxxv\" (UniqueName: \"kubernetes.io/projected/03e58c51-820b-4fae-9bbd-7f72624b2732-kube-api-access-svxxv\") pod \"03e58c51-820b-4fae-9bbd-7f72624b2732\" (UID: \"03e58c51-820b-4fae-9bbd-7f72624b2732\") " Jul 24 17:27:20.481463 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:20.481442 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e58c51-820b-4fae-9bbd-7f72624b2732-kube-api-access-svxxv" (OuterVolumeSpecName: "kube-api-access-svxxv") pod "03e58c51-820b-4fae-9bbd-7f72624b2732" (UID: "03e58c51-820b-4fae-9bbd-7f72624b2732"). InnerVolumeSpecName "kube-api-access-svxxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:27:20.580564 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:20.580536 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svxxv\" (UniqueName: \"kubernetes.io/projected/03e58c51-820b-4fae-9bbd-7f72624b2732-kube-api-access-svxxv\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:21.228943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.228913 2574 generic.go:356] "Generic (PLEG): container finished" podID="03e58c51-820b-4fae-9bbd-7f72624b2732" containerID="1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4" exitCode=0 Jul 24 17:27:21.229365 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.228955 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" Jul 24 17:27:21.229365 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.229005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" event={"ID":"03e58c51-820b-4fae-9bbd-7f72624b2732","Type":"ContainerDied","Data":"1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4"} Jul 24 17:27:21.229365 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.229049 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qwd4f" event={"ID":"03e58c51-820b-4fae-9bbd-7f72624b2732","Type":"ContainerDied","Data":"8b210395b70dc4eebbe60f2087cc0fc9982845d14753b5d1d6ef6128f12f87e2"} Jul 24 17:27:21.229365 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.229070 2574 scope.go:117] "RemoveContainer" containerID="1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4" Jul 24 17:27:21.238446 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.238430 2574 scope.go:117] "RemoveContainer" containerID="1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4" Jul 24 17:27:21.238672 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:27:21.238655 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4\": container with ID starting with 1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4 not found: ID does not exist" containerID="1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4" Jul 24 17:27:21.238718 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.238680 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4"} err="failed to get container status \"1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4\": rpc error: code = NotFound desc = could not find container \"1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4\": container with ID starting with 1b9d35be36de9dd10acd7dbef6676137f33f3b5caf539a197805d5b094208df4 not found: ID does not exist" Jul 24 17:27:21.253277 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.253255 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qwd4f"] Jul 24 17:27:21.257737 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.257703 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qwd4f"] Jul 24 17:27:21.800498 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:21.800466 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03e58c51-820b-4fae-9bbd-7f72624b2732" path="/var/lib/kubelet/pods/03e58c51-820b-4fae-9bbd-7f72624b2732/volumes" Jul 24 17:27:23.725367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:23.725340 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:27:23.726189 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:23.726168 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:27:28.091123 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:28.091085 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8txjb" Jul 24 17:27:28.092563 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:28.091154 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8txjb" Jul 24 17:27:28.113073 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:28.113044 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8txjb" Jul 24 17:27:28.284007 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:28.283977 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8txjb" Jul 24 17:27:32.185078 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.185039 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82"] Jul 24 17:27:32.185450 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.185364 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e58c51-820b-4fae-9bbd-7f72624b2732" containerName="registry-server" Jul 24 17:27:32.185450 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.185389 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e58c51-820b-4fae-9bbd-7f72624b2732" containerName="registry-server" Jul 24 17:27:32.185450 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.185448 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e58c51-820b-4fae-9bbd-7f72624b2732" containerName="registry-server" Jul 24 17:27:32.187819 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.187801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.190660 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.190644 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jf8zb\"" Jul 24 17:27:32.196512 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.196485 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82"] Jul 24 17:27:32.277625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.277585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.277838 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.277669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.277838 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.277688 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qbn4\" (UniqueName: \"kubernetes.io/projected/054a5c99-1bec-474b-9ae4-947e7467f0ed-kube-api-access-6qbn4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.378505 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.378472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.378505 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.378508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qbn4\" (UniqueName: \"kubernetes.io/projected/054a5c99-1bec-474b-9ae4-947e7467f0ed-kube-api-access-6qbn4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.378777 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.378549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.378955 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.378935 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.379018 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.378980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.387617 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.387587 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qbn4\" (UniqueName: \"kubernetes.io/projected/054a5c99-1bec-474b-9ae4-947e7467f0ed-kube-api-access-6qbn4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.498538 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.498504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:32.629779 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.629589 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82"] Jul 24 17:27:32.632786 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:32.632720 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054a5c99_1bec_474b_9ae4_947e7467f0ed.slice/crio-6e8f78f591b3ceb1cd525f1dc31bf8d95a1505df44868a1917c1086b01c1b562 WatchSource:0}: Error finding container 6e8f78f591b3ceb1cd525f1dc31bf8d95a1505df44868a1917c1086b01c1b562: Status 404 returned error can't find the container with id 6e8f78f591b3ceb1cd525f1dc31bf8d95a1505df44868a1917c1086b01c1b562 Jul 24 17:27:32.782047 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.781976 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4"] Jul 24 17:27:32.784511 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.784485 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:32.793924 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.793905 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4"] Jul 24 17:27:32.883471 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.883441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:32.883681 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.883494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkmp\" (UniqueName: \"kubernetes.io/projected/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-kube-api-access-2qkmp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:32.883792 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.883681 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:32.984786 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.984746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:32.984981 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.984797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:32.984981 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.984826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkmp\" (UniqueName: \"kubernetes.io/projected/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-kube-api-access-2qkmp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:32.985188 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.985167 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:32.985245 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.985185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:32.994364 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:32.994332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkmp\" (UniqueName: \"kubernetes.io/projected/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-kube-api-access-2qkmp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:33.093360 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.093290 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:33.224954 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.224931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4"] Jul 24 17:27:33.226666 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:33.226644 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9f2fe9_d2fb_44c7_a847_1960e1be2f64.slice/crio-b643e8c7152e409d70bc5fdfd6067e192dbca7d80f23a0e0a74bcf407b413447 WatchSource:0}: Error finding container b643e8c7152e409d70bc5fdfd6067e192dbca7d80f23a0e0a74bcf407b413447: Status 404 returned error can't find the container with id b643e8c7152e409d70bc5fdfd6067e192dbca7d80f23a0e0a74bcf407b413447 Jul 24 17:27:33.287004 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.286973 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" event={"ID":"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64","Type":"ContainerStarted","Data":"b643e8c7152e409d70bc5fdfd6067e192dbca7d80f23a0e0a74bcf407b413447"} Jul 24 17:27:33.288452 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.288430 2574 generic.go:356] "Generic (PLEG): container finished" podID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerID="6a4912bb17a2b0fc1756a9db51569862dad3b1db9c2c446a80debeee01077c9c" exitCode=0 Jul 24 17:27:33.288552 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.288497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" event={"ID":"054a5c99-1bec-474b-9ae4-947e7467f0ed","Type":"ContainerDied","Data":"6a4912bb17a2b0fc1756a9db51569862dad3b1db9c2c446a80debeee01077c9c"} Jul 24 17:27:33.288610 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.288547 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" event={"ID":"054a5c99-1bec-474b-9ae4-947e7467f0ed","Type":"ContainerStarted","Data":"6e8f78f591b3ceb1cd525f1dc31bf8d95a1505df44868a1917c1086b01c1b562"} Jul 24 17:27:33.383864 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.383805 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm"] Jul 24 17:27:33.386018 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.386003 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.395000 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.394980 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm"] Jul 24 17:27:33.488579 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.488545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.488716 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.488586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28557\" (UniqueName: \"kubernetes.io/projected/674513fc-7288-40ee-9a40-2747ac8f5852-kube-api-access-28557\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.488716 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.488700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.589445 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.589410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.589610 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.589452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28557\" (UniqueName: \"kubernetes.io/projected/674513fc-7288-40ee-9a40-2747ac8f5852-kube-api-access-28557\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.589610 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.589512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.589834 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.589813 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.589891 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.589836 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.598883 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.598861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28557\" (UniqueName: \"kubernetes.io/projected/674513fc-7288-40ee-9a40-2747ac8f5852-kube-api-access-28557\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.713126 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.713085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:33.785252 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.785225 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8"] Jul 24 17:27:33.789141 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.789120 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:33.804269 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.804211 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8"] Jul 24 17:27:33.891544 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.891518 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm"] Jul 24 17:27:33.891896 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.891870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:33.891964 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.891923 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:33.892028 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.892003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc2dv\" (UniqueName: \"kubernetes.io/projected/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-kube-api-access-jc2dv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:33.927483 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:33.927458 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674513fc_7288_40ee_9a40_2747ac8f5852.slice/crio-1a777849c1f60306de4ce2234be3da941b825ef928d416b6ed0e343ca1c48d5e WatchSource:0}: Error finding container 1a777849c1f60306de4ce2234be3da941b825ef928d416b6ed0e343ca1c48d5e: Status 404 returned error can't find the container with id 1a777849c1f60306de4ce2234be3da941b825ef928d416b6ed0e343ca1c48d5e Jul 24 17:27:33.993253 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.993225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc2dv\" (UniqueName: \"kubernetes.io/projected/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-kube-api-access-jc2dv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:33.993397 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.993312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:33.993397 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.993346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:33.993657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.993638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:33.993720 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:33.993674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:34.001688 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.001666 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc2dv\" (UniqueName: \"kubernetes.io/projected/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-kube-api-access-jc2dv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:34.106562 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.106532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:34.240319 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.240244 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8"] Jul 24 17:27:34.243302 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:34.243274 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400c4845_05ff_4d9d_8a9a_a35b388ad2f6.slice/crio-dccc1de2a556b7454823517037edb6728a2b0418d1faf922d862c46d538cfa75 WatchSource:0}: Error finding container dccc1de2a556b7454823517037edb6728a2b0418d1faf922d862c46d538cfa75: Status 404 returned error can't find the container with id dccc1de2a556b7454823517037edb6728a2b0418d1faf922d862c46d538cfa75 Jul 24 17:27:34.295448 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.295410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" event={"ID":"400c4845-05ff-4d9d-8a9a-a35b388ad2f6","Type":"ContainerStarted","Data":"dccc1de2a556b7454823517037edb6728a2b0418d1faf922d862c46d538cfa75"} Jul 24 17:27:34.296866 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.296842 2574 generic.go:356] "Generic (PLEG): container finished" podID="674513fc-7288-40ee-9a40-2747ac8f5852" containerID="33768e246ebbbf18217a0d050e09439e6e82da31ddba44e99ae8872d5169df90" exitCode=0 Jul 24 17:27:34.296969 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.296936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" event={"ID":"674513fc-7288-40ee-9a40-2747ac8f5852","Type":"ContainerDied","Data":"33768e246ebbbf18217a0d050e09439e6e82da31ddba44e99ae8872d5169df90"} Jul 24 17:27:34.297018 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.296981 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" event={"ID":"674513fc-7288-40ee-9a40-2747ac8f5852","Type":"ContainerStarted","Data":"1a777849c1f60306de4ce2234be3da941b825ef928d416b6ed0e343ca1c48d5e"} Jul 24 17:27:34.298706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.298682 2574 generic.go:356] "Generic (PLEG): container finished" podID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerID="025b81144c18ad0058ebb43bf6cdff098775f433dbd96c25e8831a384a95a9e1" exitCode=0 Jul 24 17:27:34.298823 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.298768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" event={"ID":"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64","Type":"ContainerDied","Data":"025b81144c18ad0058ebb43bf6cdff098775f433dbd96c25e8831a384a95a9e1"} Jul 24 17:27:34.301227 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.301192 2574 generic.go:356] "Generic (PLEG): container finished" podID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerID="0b38b5d50e36393cd53e1cafb949715f7415e8e27007cfb25bbd25c22d8353aa" exitCode=0 Jul 24 17:27:34.301296 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:34.301237 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" event={"ID":"054a5c99-1bec-474b-9ae4-947e7467f0ed","Type":"ContainerDied","Data":"0b38b5d50e36393cd53e1cafb949715f7415e8e27007cfb25bbd25c22d8353aa"} Jul 24 17:27:35.308022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:35.307991 2574 generic.go:356] "Generic (PLEG): container finished" podID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerID="93d584536265a1dcddc263e88da53e4296277d4fc6852e3e8f9feac9d366bc8a" exitCode=0 Jul 24 17:27:35.308403 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:35.308068 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" event={"ID":"400c4845-05ff-4d9d-8a9a-a35b388ad2f6","Type":"ContainerDied","Data":"93d584536265a1dcddc263e88da53e4296277d4fc6852e3e8f9feac9d366bc8a"} Jul 24 17:27:35.308638 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:35.308621 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:27:35.310021 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:35.309992 2574 generic.go:356] "Generic (PLEG): container finished" podID="674513fc-7288-40ee-9a40-2747ac8f5852" containerID="c68e2473930f77b17fcee50707f5ca57ebe8d53a81356c64a834723b81efdd77" exitCode=0 Jul 24 17:27:35.310099 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:35.310079 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" event={"ID":"674513fc-7288-40ee-9a40-2747ac8f5852","Type":"ContainerDied","Data":"c68e2473930f77b17fcee50707f5ca57ebe8d53a81356c64a834723b81efdd77"} Jul 24 17:27:35.312284 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:35.312264 2574 generic.go:356] "Generic (PLEG): container finished" podID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerID="403d078ca13f48163509aac46199dc9f91b18e3a1466aded55f167b5373044c8" exitCode=0 Jul 24 17:27:35.312376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:35.312358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" event={"ID":"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64","Type":"ContainerDied","Data":"403d078ca13f48163509aac46199dc9f91b18e3a1466aded55f167b5373044c8"} Jul 24 17:27:35.315050 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:35.315029 2574 generic.go:356] "Generic (PLEG): container finished" podID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerID="5fb94e3d057cfdef1dcded68d81ed06ef13b76591aa2dc524cb362e89f6c442d" exitCode=0 Jul 24 17:27:35.315150 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:35.315092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" event={"ID":"054a5c99-1bec-474b-9ae4-947e7467f0ed","Type":"ContainerDied","Data":"5fb94e3d057cfdef1dcded68d81ed06ef13b76591aa2dc524cb362e89f6c442d"} Jul 24 17:27:36.322110 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.322071 2574 generic.go:356] "Generic (PLEG): container finished" podID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerID="fff7b6603243255b7640a0b6af04ca1d341c9090edeb22470a165b9a7ba5ed9a" exitCode=0 Jul 24 17:27:36.322533 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.322172 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" event={"ID":"400c4845-05ff-4d9d-8a9a-a35b388ad2f6","Type":"ContainerDied","Data":"fff7b6603243255b7640a0b6af04ca1d341c9090edeb22470a165b9a7ba5ed9a"} Jul 24 17:27:36.324384 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.324362 2574 generic.go:356] "Generic (PLEG): container finished" podID="674513fc-7288-40ee-9a40-2747ac8f5852" containerID="93c4df0795711a8cef0185440d8695394a65279e4029a960431cf083fbaae8ee" exitCode=0 Jul 24 17:27:36.324494 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.324456 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" event={"ID":"674513fc-7288-40ee-9a40-2747ac8f5852","Type":"ContainerDied","Data":"93c4df0795711a8cef0185440d8695394a65279e4029a960431cf083fbaae8ee"} Jul 24 17:27:36.326774 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.326754 2574 generic.go:356] "Generic (PLEG): container finished" podID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerID="c3b20ee45733be58001916f31252146f4dd45d88a49dd3fb2697a14bfa62e309" exitCode=0 Jul 24 17:27:36.326874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.326813 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" event={"ID":"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64","Type":"ContainerDied","Data":"c3b20ee45733be58001916f31252146f4dd45d88a49dd3fb2697a14bfa62e309"} Jul 24 17:27:36.541833 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.541812 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:36.615791 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.615708 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qbn4\" (UniqueName: \"kubernetes.io/projected/054a5c99-1bec-474b-9ae4-947e7467f0ed-kube-api-access-6qbn4\") pod \"054a5c99-1bec-474b-9ae4-947e7467f0ed\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " Jul 24 17:27:36.615950 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.615827 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-bundle\") pod \"054a5c99-1bec-474b-9ae4-947e7467f0ed\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " Jul 24 17:27:36.615950 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.615846 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-util\") pod \"054a5c99-1bec-474b-9ae4-947e7467f0ed\" (UID: \"054a5c99-1bec-474b-9ae4-947e7467f0ed\") " Jul 24 17:27:36.616389 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.616353 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-bundle" (OuterVolumeSpecName: "bundle") pod "054a5c99-1bec-474b-9ae4-947e7467f0ed" (UID: "054a5c99-1bec-474b-9ae4-947e7467f0ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:27:36.617940 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.617915 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054a5c99-1bec-474b-9ae4-947e7467f0ed-kube-api-access-6qbn4" (OuterVolumeSpecName: "kube-api-access-6qbn4") pod "054a5c99-1bec-474b-9ae4-947e7467f0ed" (UID: "054a5c99-1bec-474b-9ae4-947e7467f0ed"). InnerVolumeSpecName "kube-api-access-6qbn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:27:36.621528 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.621507 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-util" (OuterVolumeSpecName: "util") pod "054a5c99-1bec-474b-9ae4-947e7467f0ed" (UID: "054a5c99-1bec-474b-9ae4-947e7467f0ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:27:36.717020 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.716993 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qbn4\" (UniqueName: \"kubernetes.io/projected/054a5c99-1bec-474b-9ae4-947e7467f0ed-kube-api-access-6qbn4\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:36.717020 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.717015 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-bundle\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:36.717020 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:36.717024 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/054a5c99-1bec-474b-9ae4-947e7467f0ed-util\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:37.336310 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.336281 2574 generic.go:356] "Generic (PLEG): container finished" podID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerID="b3e1c8fdadcb469bbb06c16850869f896b9bf1233ff384f36a2dd60dedd7dbeb" exitCode=0 Jul 24 17:27:37.336743 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.336334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" event={"ID":"400c4845-05ff-4d9d-8a9a-a35b388ad2f6","Type":"ContainerDied","Data":"b3e1c8fdadcb469bbb06c16850869f896b9bf1233ff384f36a2dd60dedd7dbeb"} Jul 24 17:27:37.338284 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.338266 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" Jul 24 17:27:37.338336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.338300 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82" event={"ID":"054a5c99-1bec-474b-9ae4-947e7467f0ed","Type":"ContainerDied","Data":"6e8f78f591b3ceb1cd525f1dc31bf8d95a1505df44868a1917c1086b01c1b562"} Jul 24 17:27:37.338336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.338333 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e8f78f591b3ceb1cd525f1dc31bf8d95a1505df44868a1917c1086b01c1b562" Jul 24 17:27:37.476426 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.476404 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:37.505981 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.505960 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:37.522734 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.522710 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-bundle\") pod \"674513fc-7288-40ee-9a40-2747ac8f5852\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " Jul 24 17:27:37.522860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.522787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-util\") pod \"674513fc-7288-40ee-9a40-2747ac8f5852\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " Jul 24 17:27:37.522860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.522833 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28557\" (UniqueName: \"kubernetes.io/projected/674513fc-7288-40ee-9a40-2747ac8f5852-kube-api-access-28557\") pod \"674513fc-7288-40ee-9a40-2747ac8f5852\" (UID: \"674513fc-7288-40ee-9a40-2747ac8f5852\") " Jul 24 17:27:37.523244 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.523178 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-bundle" (OuterVolumeSpecName: "bundle") pod "674513fc-7288-40ee-9a40-2747ac8f5852" (UID: "674513fc-7288-40ee-9a40-2747ac8f5852"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:27:37.526026 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.525990 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674513fc-7288-40ee-9a40-2747ac8f5852-kube-api-access-28557" (OuterVolumeSpecName: "kube-api-access-28557") pod "674513fc-7288-40ee-9a40-2747ac8f5852" (UID: "674513fc-7288-40ee-9a40-2747ac8f5852"). InnerVolumeSpecName "kube-api-access-28557". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:27:37.530111 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.530087 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-util" (OuterVolumeSpecName: "util") pod "674513fc-7288-40ee-9a40-2747ac8f5852" (UID: "674513fc-7288-40ee-9a40-2747ac8f5852"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:27:37.623783 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.623696 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-util\") pod \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " Jul 24 17:27:37.623783 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.623746 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-bundle\") pod \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " Jul 24 17:27:37.623956 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.623796 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qkmp\" (UniqueName: \"kubernetes.io/projected/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-kube-api-access-2qkmp\") pod \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\" (UID: \"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64\") " Jul 24 17:27:37.624010 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.623994 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-bundle\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:37.624046 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.624017 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/674513fc-7288-40ee-9a40-2747ac8f5852-util\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:37.624046 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.624030 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28557\" (UniqueName: \"kubernetes.io/projected/674513fc-7288-40ee-9a40-2747ac8f5852-kube-api-access-28557\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:37.624235 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.624211 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-bundle" (OuterVolumeSpecName: "bundle") pod "bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" (UID: "bc9f2fe9-d2fb-44c7-a847-1960e1be2f64"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:27:37.626022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.626002 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-kube-api-access-2qkmp" (OuterVolumeSpecName: "kube-api-access-2qkmp") pod "bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" (UID: "bc9f2fe9-d2fb-44c7-a847-1960e1be2f64"). InnerVolumeSpecName "kube-api-access-2qkmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:27:37.629123 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.629100 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-util" (OuterVolumeSpecName: "util") pod "bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" (UID: "bc9f2fe9-d2fb-44c7-a847-1960e1be2f64"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:27:37.724709 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.724680 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-util\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:37.724709 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.724705 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-bundle\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:37.724892 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.724716 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkmp\" (UniqueName: \"kubernetes.io/projected/bc9f2fe9-d2fb-44c7-a847-1960e1be2f64-kube-api-access-2qkmp\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:37.965223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965185 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b45fff8ff-86jrg"] Jul 24 17:27:37.965560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965546 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerName="extract" Jul 24 17:27:37.965613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965562 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerName="extract" Jul 24 17:27:37.965613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965573 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerName="pull" Jul 24 17:27:37.965613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965578 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerName="pull" Jul 24 17:27:37.965613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965586 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674513fc-7288-40ee-9a40-2747ac8f5852" containerName="pull" Jul 24 17:27:37.965613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965592 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="674513fc-7288-40ee-9a40-2747ac8f5852" containerName="pull" Jul 24 17:27:37.965613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965600 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerName="pull" Jul 24 17:27:37.965613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965604 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerName="pull" Jul 24 17:27:37.965613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965611 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerName="util" Jul 24 17:27:37.965613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965617 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerName="util" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965628 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerName="util" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965633 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerName="util" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965639 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674513fc-7288-40ee-9a40-2747ac8f5852" containerName="extract" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965644 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="674513fc-7288-40ee-9a40-2747ac8f5852" containerName="extract" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965648 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerName="extract" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965653 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerName="extract" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965664 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674513fc-7288-40ee-9a40-2747ac8f5852" containerName="util" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965669 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="674513fc-7288-40ee-9a40-2747ac8f5852" containerName="util" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965738 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc9f2fe9-d2fb-44c7-a847-1960e1be2f64" containerName="extract" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965748 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="674513fc-7288-40ee-9a40-2747ac8f5852" containerName="extract" Jul 24 17:27:37.965973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.965757 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="054a5c99-1bec-474b-9ae4-947e7467f0ed" containerName="extract" Jul 24 17:27:37.967930 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.967915 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:37.982999 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:37.982972 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b45fff8ff-86jrg"] Jul 24 17:27:38.027782 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.027749 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-trusted-ca-bundle\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.027941 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.027790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-oauth-serving-cert\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.027941 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.027807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-console-oauth-config\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.027941 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.027860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2m4\" (UniqueName: \"kubernetes.io/projected/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-kube-api-access-nh2m4\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.027941 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.027876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-service-ca\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.028125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.027946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-console-serving-cert\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.028125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.027967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-console-config\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.128657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.128627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-trusted-ca-bundle\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.128657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.128662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-oauth-serving-cert\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.128910 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.128681 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-console-oauth-config\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.128910 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.128702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2m4\" (UniqueName: \"kubernetes.io/projected/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-kube-api-access-nh2m4\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.128910 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.128718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-service-ca\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.128910 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.128798 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-console-serving-cert\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.129114 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.129008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-console-config\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.129493 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.129460 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-oauth-serving-cert\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.129596 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.129496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-service-ca\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.129596 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.129562 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-trusted-ca-bundle\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.129678 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.129598 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-console-config\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.131711 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.131692 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-console-oauth-config\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.131780 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.131694 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-console-serving-cert\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.137946 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.137927 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2m4\" (UniqueName: \"kubernetes.io/projected/2a86c2b9-0af0-4fde-a37f-f61e949cf9c7-kube-api-access-nh2m4\") pod \"console-5b45fff8ff-86jrg\" (UID: \"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7\") " pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.277638 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.277559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:38.347267 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.347235 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" Jul 24 17:27:38.347267 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.347258 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm" event={"ID":"674513fc-7288-40ee-9a40-2747ac8f5852","Type":"ContainerDied","Data":"1a777849c1f60306de4ce2234be3da941b825ef928d416b6ed0e343ca1c48d5e"} Jul 24 17:27:38.347667 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.347301 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a777849c1f60306de4ce2234be3da941b825ef928d416b6ed0e343ca1c48d5e" Jul 24 17:27:38.349480 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.349459 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" Jul 24 17:27:38.349613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.349476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4" event={"ID":"bc9f2fe9-d2fb-44c7-a847-1960e1be2f64","Type":"ContainerDied","Data":"b643e8c7152e409d70bc5fdfd6067e192dbca7d80f23a0e0a74bcf407b413447"} Jul 24 17:27:38.349613 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.349530 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b643e8c7152e409d70bc5fdfd6067e192dbca7d80f23a0e0a74bcf407b413447" Jul 24 17:27:38.425865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.425834 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b45fff8ff-86jrg"] Jul 24 17:27:38.428758 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:38.428717 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a86c2b9_0af0_4fde_a37f_f61e949cf9c7.slice/crio-0b2187cc3c2b39d532ba0f4457845edd50abb7ebb309dda89493612abf4fe507 WatchSource:0}: Error finding container 0b2187cc3c2b39d532ba0f4457845edd50abb7ebb309dda89493612abf4fe507: Status 404 returned error can't find the container with id 0b2187cc3c2b39d532ba0f4457845edd50abb7ebb309dda89493612abf4fe507 Jul 24 17:27:38.502309 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.502287 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:38.532415 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.532324 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc2dv\" (UniqueName: \"kubernetes.io/projected/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-kube-api-access-jc2dv\") pod \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " Jul 24 17:27:38.532565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.532443 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-bundle\") pod \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " Jul 24 17:27:38.532565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.532504 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-util\") pod \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\" (UID: \"400c4845-05ff-4d9d-8a9a-a35b388ad2f6\") " Jul 24 17:27:38.532985 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.532949 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-bundle" (OuterVolumeSpecName: "bundle") pod "400c4845-05ff-4d9d-8a9a-a35b388ad2f6" (UID: "400c4845-05ff-4d9d-8a9a-a35b388ad2f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:27:38.534654 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.534627 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-kube-api-access-jc2dv" (OuterVolumeSpecName: "kube-api-access-jc2dv") pod "400c4845-05ff-4d9d-8a9a-a35b388ad2f6" (UID: "400c4845-05ff-4d9d-8a9a-a35b388ad2f6"). InnerVolumeSpecName "kube-api-access-jc2dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:27:38.539342 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.539318 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-util" (OuterVolumeSpecName: "util") pod "400c4845-05ff-4d9d-8a9a-a35b388ad2f6" (UID: "400c4845-05ff-4d9d-8a9a-a35b388ad2f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:27:38.634021 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.633990 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-bundle\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:38.634021 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.634015 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-util\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:38.634021 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:38.634025 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc2dv\" (UniqueName: \"kubernetes.io/projected/400c4845-05ff-4d9d-8a9a-a35b388ad2f6-kube-api-access-jc2dv\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:27:39.356333 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:39.356290 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b45fff8ff-86jrg" event={"ID":"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7","Type":"ContainerStarted","Data":"541402987f976607d18425aaf1a7c8b28082a91800d92a228eea3113d0fc6a01"} Jul 24 17:27:39.356764 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:39.356350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b45fff8ff-86jrg" event={"ID":"2a86c2b9-0af0-4fde-a37f-f61e949cf9c7","Type":"ContainerStarted","Data":"0b2187cc3c2b39d532ba0f4457845edd50abb7ebb309dda89493612abf4fe507"} Jul 24 17:27:39.358149 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:39.358116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" event={"ID":"400c4845-05ff-4d9d-8a9a-a35b388ad2f6","Type":"ContainerDied","Data":"dccc1de2a556b7454823517037edb6728a2b0418d1faf922d862c46d538cfa75"} Jul 24 17:27:39.358264 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:39.358154 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8" Jul 24 17:27:39.358264 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:39.358168 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dccc1de2a556b7454823517037edb6728a2b0418d1faf922d862c46d538cfa75" Jul 24 17:27:39.373989 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:39.373950 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b45fff8ff-86jrg" podStartSLOduration=2.3739381 podStartE2EDuration="2.3739381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:27:39.37364782 +0000 UTC m=+616.039222456" watchObservedRunningTime="2026-07-24 17:27:39.3739381 +0000 UTC m=+616.039512737" Jul 24 17:27:48.277682 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.277652 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:48.278091 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.277716 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:48.282492 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.282469 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:48.405864 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.405837 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b45fff8ff-86jrg" Jul 24 17:27:48.464208 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.461540 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9"] Jul 24 17:27:48.464208 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.462355 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerName="pull" Jul 24 17:27:48.464208 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.462374 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerName="pull" Jul 24 17:27:48.464208 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.462437 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerName="extract" Jul 24 17:27:48.464208 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.462446 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerName="extract" Jul 24 17:27:48.464208 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.462474 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerName="util" Jul 24 17:27:48.464208 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.462489 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerName="util" Jul 24 17:27:48.464208 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.462633 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="400c4845-05ff-4d9d-8a9a-a35b388ad2f6" containerName="extract" Jul 24 17:27:48.468867 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.468839 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" Jul 24 17:27:48.472810 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.472779 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kkpsg\"" Jul 24 17:27:48.472985 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.472880 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 17:27:48.472985 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.472789 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-654c7856b8-w5kfn"] Jul 24 17:27:48.476160 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.476134 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9"] Jul 24 17:27:48.516880 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.516843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8t8\" (UniqueName: \"kubernetes.io/projected/22a0600b-eaa2-4f90-b2f9-bf9b02571d6b-kube-api-access-vd8t8\") pod \"dns-operator-controller-manager-864dd78c4c-hzpg9\" (UID: \"22a0600b-eaa2-4f90-b2f9-bf9b02571d6b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" Jul 24 17:27:48.618160 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.618059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8t8\" (UniqueName: \"kubernetes.io/projected/22a0600b-eaa2-4f90-b2f9-bf9b02571d6b-kube-api-access-vd8t8\") pod \"dns-operator-controller-manager-864dd78c4c-hzpg9\" (UID: \"22a0600b-eaa2-4f90-b2f9-bf9b02571d6b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" Jul 24 17:27:48.631148 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.631121 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8t8\" (UniqueName: \"kubernetes.io/projected/22a0600b-eaa2-4f90-b2f9-bf9b02571d6b-kube-api-access-vd8t8\") pod \"dns-operator-controller-manager-864dd78c4c-hzpg9\" (UID: \"22a0600b-eaa2-4f90-b2f9-bf9b02571d6b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" Jul 24 17:27:48.790262 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.790224 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" Jul 24 17:27:48.929316 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:48.929287 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9"] Jul 24 17:27:48.931164 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:48.931136 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a0600b_eaa2_4f90_b2f9_bf9b02571d6b.slice/crio-89c7b8ebc8adcbf8eb6e28fb4727cc025f4571a0bb714650ea5b76b45d5beae4 WatchSource:0}: Error finding container 89c7b8ebc8adcbf8eb6e28fb4727cc025f4571a0bb714650ea5b76b45d5beae4: Status 404 returned error can't find the container with id 89c7b8ebc8adcbf8eb6e28fb4727cc025f4571a0bb714650ea5b76b45d5beae4 Jul 24 17:27:49.408587 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:49.408552 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" event={"ID":"22a0600b-eaa2-4f90-b2f9-bf9b02571d6b","Type":"ContainerStarted","Data":"89c7b8ebc8adcbf8eb6e28fb4727cc025f4571a0bb714650ea5b76b45d5beae4"} Jul 24 17:27:51.422001 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:51.421970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" event={"ID":"22a0600b-eaa2-4f90-b2f9-bf9b02571d6b","Type":"ContainerStarted","Data":"021a6336b6d5578cd586b7f7dff471a995e6e9834a44d6e3c9299ab4192a3c8b"} Jul 24 17:27:51.422424 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:51.422060 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" Jul 24 17:27:51.444657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:51.444609 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" podStartSLOduration=1.892211219 podStartE2EDuration="3.444594582s" totalImagesPullingTime="1.552383363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:27:51.443053912 +0000 UTC m=+628.108628549" watchObservedRunningTime="2026-07-24 17:27:51.444594582 +0000 UTC m=+628.110169219" Jul 24 17:27:53.741608 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.741577 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2"] Jul 24 17:27:53.747596 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.747577 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:27:53.750854 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.750831 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-csc7m\"" Jul 24 17:27:53.757593 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.757573 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2"] Jul 24 17:27:53.865519 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.865484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntfc\" (UniqueName: \"kubernetes.io/projected/1c662871-978a-4f4c-89fa-959965d6b48c-kube-api-access-cntfc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" (UID: \"1c662871-978a-4f4c-89fa-959965d6b48c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:27:53.865743 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.865549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c662871-978a-4f4c-89fa-959965d6b48c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" (UID: \"1c662871-978a-4f4c-89fa-959965d6b48c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:27:53.966435 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.966399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntfc\" (UniqueName: \"kubernetes.io/projected/1c662871-978a-4f4c-89fa-959965d6b48c-kube-api-access-cntfc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" (UID: \"1c662871-978a-4f4c-89fa-959965d6b48c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:27:53.966657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.966452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c662871-978a-4f4c-89fa-959965d6b48c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" (UID: \"1c662871-978a-4f4c-89fa-959965d6b48c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:27:53.966889 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.966867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c662871-978a-4f4c-89fa-959965d6b48c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" (UID: \"1c662871-978a-4f4c-89fa-959965d6b48c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:27:53.977482 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:53.977459 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntfc\" (UniqueName: \"kubernetes.io/projected/1c662871-978a-4f4c-89fa-959965d6b48c-kube-api-access-cntfc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" (UID: \"1c662871-978a-4f4c-89fa-959965d6b48c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:27:54.059230 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:54.059144 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:27:54.203838 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:54.203760 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2"] Jul 24 17:27:54.206299 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:54.206270 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c662871_978a_4f4c_89fa_959965d6b48c.slice/crio-34063e99a453b23ccc6b40c17050f1f78250f510f6ab59855b130b270ddcd11a WatchSource:0}: Error finding container 34063e99a453b23ccc6b40c17050f1f78250f510f6ab59855b130b270ddcd11a: Status 404 returned error can't find the container with id 34063e99a453b23ccc6b40c17050f1f78250f510f6ab59855b130b270ddcd11a Jul 24 17:27:54.437747 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:54.437629 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" event={"ID":"1c662871-978a-4f4c-89fa-959965d6b48c","Type":"ContainerStarted","Data":"34063e99a453b23ccc6b40c17050f1f78250f510f6ab59855b130b270ddcd11a"} Jul 24 17:27:57.174449 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:57.174409 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg"] Jul 24 17:27:57.178807 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:57.178786 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" Jul 24 17:27:57.182060 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:57.182035 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-prlcp\"" Jul 24 17:27:57.190860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:57.190809 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg"] Jul 24 17:27:57.297692 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:57.297659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khpkv\" (UniqueName: \"kubernetes.io/projected/a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2-kube-api-access-khpkv\") pod \"authorino-operator-75cc8d5cb7-x2tdg\" (UID: \"a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" Jul 24 17:27:57.398798 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:57.398758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khpkv\" (UniqueName: \"kubernetes.io/projected/a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2-kube-api-access-khpkv\") pod \"authorino-operator-75cc8d5cb7-x2tdg\" (UID: \"a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" Jul 24 17:27:57.414979 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:57.414947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khpkv\" (UniqueName: \"kubernetes.io/projected/a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2-kube-api-access-khpkv\") pod \"authorino-operator-75cc8d5cb7-x2tdg\" (UID: \"a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" Jul 24 17:27:57.492087 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:57.492051 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" Jul 24 17:27:58.286058 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:58.286038 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg"] Jul 24 17:27:58.287783 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:27:58.287758 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f1bd49_a0f4_4af7_8d9d_d3bf949b92e2.slice/crio-932d93c30886687e4615777f27d1ef38c5758c28d30e9afdeb0514a39860a6aa WatchSource:0}: Error finding container 932d93c30886687e4615777f27d1ef38c5758c28d30e9afdeb0514a39860a6aa: Status 404 returned error can't find the container with id 932d93c30886687e4615777f27d1ef38c5758c28d30e9afdeb0514a39860a6aa Jul 24 17:27:58.462754 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:58.462693 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" event={"ID":"1c662871-978a-4f4c-89fa-959965d6b48c","Type":"ContainerStarted","Data":"8139c197a4ee2bf5d8405891e5f0b318c21ad31fe9567083053ac9e595af121a"} Jul 24 17:27:58.462925 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:58.462787 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:27:58.464080 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:58.464057 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" event={"ID":"a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2","Type":"ContainerStarted","Data":"932d93c30886687e4615777f27d1ef38c5758c28d30e9afdeb0514a39860a6aa"} Jul 24 17:27:58.485932 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:27:58.485881 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" podStartSLOduration=1.4866544099999999 podStartE2EDuration="5.48586136s" totalImagesPullingTime="3.99920695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:27:58.484078247 +0000 UTC m=+635.149652897" watchObservedRunningTime="2026-07-24 17:27:58.48586136 +0000 UTC m=+635.151435996" Jul 24 17:28:00.099534 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:00.099502 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2"] Jul 24 17:28:00.110532 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:00.110506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:00.111637 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:00.111600 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2"] Jul 24 17:28:00.113481 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:00.113459 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ddxsj\"" Jul 24 17:28:00.222954 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:00.222915 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xrmg\" (UniqueName: \"kubernetes.io/projected/ab680537-ed21-4677-9889-5a6a578a20b3-kube-api-access-5xrmg\") pod \"limitador-operator-controller-manager-859d7b78bd-h78z2\" (UID: \"ab680537-ed21-4677-9889-5a6a578a20b3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:00.324256 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:00.324149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xrmg\" (UniqueName: \"kubernetes.io/projected/ab680537-ed21-4677-9889-5a6a578a20b3-kube-api-access-5xrmg\") pod \"limitador-operator-controller-manager-859d7b78bd-h78z2\" (UID: \"ab680537-ed21-4677-9889-5a6a578a20b3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:00.336502 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:00.336459 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xrmg\" (UniqueName: \"kubernetes.io/projected/ab680537-ed21-4677-9889-5a6a578a20b3-kube-api-access-5xrmg\") pod \"limitador-operator-controller-manager-859d7b78bd-h78z2\" (UID: \"ab680537-ed21-4677-9889-5a6a578a20b3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:00.423293 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:00.423214 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:00.581843 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:00.581815 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2"] Jul 24 17:28:00.583355 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:28:00.583321 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab680537_ed21_4677_9889_5a6a578a20b3.slice/crio-f91b6d19dfcdce33019c6bd448cceeab364a3100947cf71a4ceadc97c96f9234 WatchSource:0}: Error finding container f91b6d19dfcdce33019c6bd448cceeab364a3100947cf71a4ceadc97c96f9234: Status 404 returned error can't find the container with id f91b6d19dfcdce33019c6bd448cceeab364a3100947cf71a4ceadc97c96f9234 Jul 24 17:28:01.349642 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.349605 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q"] Jul 24 17:28:01.353252 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.353230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.356810 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.356550 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 17:28:01.356810 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.356634 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 17:28:01.356810 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.356907 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jf8zb\"" Jul 24 17:28:01.363541 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.363371 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q"] Jul 24 17:28:01.433397 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.433364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c118badb-e8bc-4a8d-95ea-ed1e95b970dc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jss8q\" (UID: \"c118badb-e8bc-4a8d-95ea-ed1e95b970dc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.433397 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.433397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c118badb-e8bc-4a8d-95ea-ed1e95b970dc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jss8q\" (UID: \"c118badb-e8bc-4a8d-95ea-ed1e95b970dc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.433600 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.433531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqkq\" (UniqueName: \"kubernetes.io/projected/c118badb-e8bc-4a8d-95ea-ed1e95b970dc-kube-api-access-lsqkq\") pod \"kuadrant-console-plugin-66b5774985-jss8q\" (UID: \"c118badb-e8bc-4a8d-95ea-ed1e95b970dc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.484562 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.484523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" event={"ID":"a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2","Type":"ContainerStarted","Data":"04c8d4b4a233f0236c7d666683d0667fb1ad23465da099ba28dace7d87a221e2"} Jul 24 17:28:01.484783 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.484738 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" Jul 24 17:28:01.486089 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.486061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" event={"ID":"ab680537-ed21-4677-9889-5a6a578a20b3","Type":"ContainerStarted","Data":"f91b6d19dfcdce33019c6bd448cceeab364a3100947cf71a4ceadc97c96f9234"} Jul 24 17:28:01.505426 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.505378 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" podStartSLOduration=2.3486331959999998 podStartE2EDuration="4.505360826s" totalImagesPullingTime="2.15672763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:01.50235703 +0000 UTC m=+638.167931680" watchObservedRunningTime="2026-07-24 17:28:01.505360826 +0000 UTC m=+638.170935531" Jul 24 17:28:01.534963 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.534929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqkq\" (UniqueName: \"kubernetes.io/projected/c118badb-e8bc-4a8d-95ea-ed1e95b970dc-kube-api-access-lsqkq\") pod \"kuadrant-console-plugin-66b5774985-jss8q\" (UID: \"c118badb-e8bc-4a8d-95ea-ed1e95b970dc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.535123 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.534984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c118badb-e8bc-4a8d-95ea-ed1e95b970dc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jss8q\" (UID: \"c118badb-e8bc-4a8d-95ea-ed1e95b970dc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.535123 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.535000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c118badb-e8bc-4a8d-95ea-ed1e95b970dc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jss8q\" (UID: \"c118badb-e8bc-4a8d-95ea-ed1e95b970dc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.535630 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.535612 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c118badb-e8bc-4a8d-95ea-ed1e95b970dc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jss8q\" (UID: \"c118badb-e8bc-4a8d-95ea-ed1e95b970dc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.537504 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.537480 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c118badb-e8bc-4a8d-95ea-ed1e95b970dc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jss8q\" (UID: \"c118badb-e8bc-4a8d-95ea-ed1e95b970dc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.544353 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.544326 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqkq\" (UniqueName: \"kubernetes.io/projected/c118badb-e8bc-4a8d-95ea-ed1e95b970dc-kube-api-access-lsqkq\") pod \"kuadrant-console-plugin-66b5774985-jss8q\" (UID: \"c118badb-e8bc-4a8d-95ea-ed1e95b970dc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:01.666246 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:01.666173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" Jul 24 17:28:02.103260 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:02.103234 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q"] Jul 24 17:28:02.104175 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:28:02.104146 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc118badb_e8bc_4a8d_95ea_ed1e95b970dc.slice/crio-d33c08d12e1246e972eb4614b0211ffefdac731793ad21fcbd2fc87a955e7c22 WatchSource:0}: Error finding container d33c08d12e1246e972eb4614b0211ffefdac731793ad21fcbd2fc87a955e7c22: Status 404 returned error can't find the container with id d33c08d12e1246e972eb4614b0211ffefdac731793ad21fcbd2fc87a955e7c22 Jul 24 17:28:02.429831 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:02.429804 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hzpg9" Jul 24 17:28:02.493673 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:02.493639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" event={"ID":"c118badb-e8bc-4a8d-95ea-ed1e95b970dc","Type":"ContainerStarted","Data":"d33c08d12e1246e972eb4614b0211ffefdac731793ad21fcbd2fc87a955e7c22"} Jul 24 17:28:02.495353 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:02.495324 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" event={"ID":"ab680537-ed21-4677-9889-5a6a578a20b3","Type":"ContainerStarted","Data":"97fd585518e429e19ac99069cc860b9a4b9cc43e67f67006c6d0b147f50f8fd4"} Jul 24 17:28:02.495620 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:02.495581 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:02.513166 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:02.513114 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" podStartSLOduration=1.102452264 podStartE2EDuration="2.513099916s" totalImagesPullingTime="1.410647652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:02.511320738 +0000 UTC m=+639.176895375" watchObservedRunningTime="2026-07-24 17:28:02.513099916 +0000 UTC m=+639.178674553" Jul 24 17:28:09.472126 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:09.472082 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:11.272073 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.272036 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2"] Jul 24 17:28:11.272569 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.272270 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" containerName="manager" containerID="cri-o://8139c197a4ee2bf5d8405891e5f0b318c21ad31fe9567083053ac9e595af121a" gracePeriod=2 Jul 24 17:28:11.275256 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:11.275221 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:11.278682 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.278644 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2"] Jul 24 17:28:11.288482 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.288458 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2"] Jul 24 17:28:11.288886 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.288692 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" containerName="manager" containerID="cri-o://97fd585518e429e19ac99069cc860b9a4b9cc43e67f67006c6d0b147f50f8fd4" gracePeriod=2 Jul 24 17:28:11.291121 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.291069 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:11.291463 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:11.291372 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:11.294539 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:11.294358 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h78z2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:11.295894 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.295855 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2"] Jul 24 17:28:11.296906 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:11.296877 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:11.298987 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.298963 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj"] Jul 24 17:28:11.299322 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:11.299282 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h78z2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:11.299457 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.299404 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" containerName="manager" Jul 24 17:28:11.299457 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.299419 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" containerName="manager" Jul 24 17:28:11.299575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.299462 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" containerName="manager" Jul 24 17:28:11.299575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.299471 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" containerName="manager" Jul 24 17:28:11.299575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.299571 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" containerName="manager" Jul 24 17:28:11.299706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.299593 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" containerName="manager" Jul 24 17:28:11.314991 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.314967 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8"] Jul 24 17:28:11.315139 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.315125 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:11.319111 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:11.319079 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h78z2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:11.319389 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.319373 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj"] Jul 24 17:28:11.319673 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.319457 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" Jul 24 17:28:11.328649 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.328592 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8"] Jul 24 17:28:11.335915 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.335870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgnjc\" (UniqueName: \"kubernetes.io/projected/c76563d8-aafb-446e-aae1-2fccbf2d4939-kube-api-access-wgnjc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkbcj\" (UID: \"c76563d8-aafb-446e-aae1-2fccbf2d4939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:11.336036 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.335935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzfmh\" (UniqueName: \"kubernetes.io/projected/2e9cb16f-80db-48b7-8953-61ff7fa9189f-kube-api-access-zzfmh\") pod \"limitador-operator-controller-manager-859d7b78bd-gjkz8\" (UID: \"2e9cb16f-80db-48b7-8953-61ff7fa9189f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" Jul 24 17:28:11.336036 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.335994 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c76563d8-aafb-446e-aae1-2fccbf2d4939-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkbcj\" (UID: \"c76563d8-aafb-446e-aae1-2fccbf2d4939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:11.343551 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:11.343519 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:11.374391 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:11.374332 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h78z2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:11.376855 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:11.376822 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:11.437363 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.437322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c76563d8-aafb-446e-aae1-2fccbf2d4939-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkbcj\" (UID: \"c76563d8-aafb-446e-aae1-2fccbf2d4939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:11.437518 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.437414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgnjc\" (UniqueName: \"kubernetes.io/projected/c76563d8-aafb-446e-aae1-2fccbf2d4939-kube-api-access-wgnjc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkbcj\" (UID: \"c76563d8-aafb-446e-aae1-2fccbf2d4939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:11.437518 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.437463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzfmh\" (UniqueName: \"kubernetes.io/projected/2e9cb16f-80db-48b7-8953-61ff7fa9189f-kube-api-access-zzfmh\") pod \"limitador-operator-controller-manager-859d7b78bd-gjkz8\" (UID: \"2e9cb16f-80db-48b7-8953-61ff7fa9189f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" Jul 24 17:28:11.437782 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.437761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c76563d8-aafb-446e-aae1-2fccbf2d4939-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkbcj\" (UID: \"c76563d8-aafb-446e-aae1-2fccbf2d4939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:11.447397 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.447370 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzfmh\" (UniqueName: \"kubernetes.io/projected/2e9cb16f-80db-48b7-8953-61ff7fa9189f-kube-api-access-zzfmh\") pod \"limitador-operator-controller-manager-859d7b78bd-gjkz8\" (UID: \"2e9cb16f-80db-48b7-8953-61ff7fa9189f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" Jul 24 17:28:11.447562 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.447530 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgnjc\" (UniqueName: \"kubernetes.io/projected/c76563d8-aafb-446e-aae1-2fccbf2d4939-kube-api-access-wgnjc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkbcj\" (UID: \"c76563d8-aafb-446e-aae1-2fccbf2d4939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:11.709882 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.709852 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:11.716030 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:11.716000 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" Jul 24 17:28:12.497836 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:12.497805 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x2tdg" Jul 24 17:28:13.498395 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:13.498337 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-654c7856b8-w5kfn" podUID="7497ef0a-c1a0-4df2-ac56-70564163d239" containerName="console" containerID="cri-o://9b939532a87dc0f8b8f6e837e5b8a696711945e573fc8cbb3631b43305543532" gracePeriod=15 Jul 24 17:28:17.255413 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:17.255370 2574 patch_prober.go:28] interesting pod/console-654c7856b8-w5kfn container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.22:8443/health\": dial tcp 10.133.0.22:8443: connect: connection refused" start-of-body= Jul 24 17:28:17.255818 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:17.255441 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-654c7856b8-w5kfn" podUID="7497ef0a-c1a0-4df2-ac56-70564163d239" containerName="console" probeResult="failure" output="Get \"https://10.133.0.22:8443/health\": dial tcp 10.133.0.22:8443: connect: connection refused" Jul 24 17:28:22.628707 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.628669 2574 generic.go:356] "Generic (PLEG): container finished" podID="7497ef0a-c1a0-4df2-ac56-70564163d239" containerID="9b939532a87dc0f8b8f6e837e5b8a696711945e573fc8cbb3631b43305543532" exitCode=0 Jul 24 17:28:22.629209 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.628778 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654c7856b8-w5kfn" event={"ID":"7497ef0a-c1a0-4df2-ac56-70564163d239","Type":"ContainerDied","Data":"9b939532a87dc0f8b8f6e837e5b8a696711945e573fc8cbb3631b43305543532"} Jul 24 17:28:22.630975 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.630880 2574 generic.go:356] "Generic (PLEG): container finished" podID="1c662871-978a-4f4c-89fa-959965d6b48c" containerID="8139c197a4ee2bf5d8405891e5f0b318c21ad31fe9567083053ac9e595af121a" exitCode=0 Jul 24 17:28:22.637077 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.636712 2574 generic.go:356] "Generic (PLEG): container finished" podID="ab680537-ed21-4677-9889-5a6a578a20b3" containerID="97fd585518e429e19ac99069cc860b9a4b9cc43e67f67006c6d0b147f50f8fd4" exitCode=0 Jul 24 17:28:22.820345 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.820321 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:28:22.822584 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.822513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8"] Jul 24 17:28:22.825903 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:28:22.825880 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9cb16f_80db_48b7_8953_61ff7fa9189f.slice/crio-6a2084afc209e36f26b6deabdf9bf9637a2ca2f8ad9abb2fc72a008a06e9a163 WatchSource:0}: Error finding container 6a2084afc209e36f26b6deabdf9bf9637a2ca2f8ad9abb2fc72a008a06e9a163: Status 404 returned error can't find the container with id 6a2084afc209e36f26b6deabdf9bf9637a2ca2f8ad9abb2fc72a008a06e9a163 Jul 24 17:28:22.832654 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.832637 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:22.839408 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.839388 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj"] Jul 24 17:28:22.840138 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.840113 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:22.843449 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:28:22.843429 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76563d8_aafb_446e_aae1_2fccbf2d4939.slice/crio-a11464812ca5ff7951cadb7a1663f9fc7771d6b569043a09dc360f3eaf2885ac WatchSource:0}: Error finding container a11464812ca5ff7951cadb7a1663f9fc7771d6b569043a09dc360f3eaf2885ac: Status 404 returned error can't find the container with id a11464812ca5ff7951cadb7a1663f9fc7771d6b569043a09dc360f3eaf2885ac Jul 24 17:28:22.847402 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:22.847247 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h78z2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:22.849810 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:22.849780 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:22.948313 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948283 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-oauth-config\") pod \"7497ef0a-c1a0-4df2-ac56-70564163d239\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " Jul 24 17:28:22.948313 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948322 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-service-ca\") pod \"7497ef0a-c1a0-4df2-ac56-70564163d239\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " Jul 24 17:28:22.948575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948358 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-trusted-ca-bundle\") pod \"7497ef0a-c1a0-4df2-ac56-70564163d239\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " Jul 24 17:28:22.948575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948373 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-oauth-serving-cert\") pod \"7497ef0a-c1a0-4df2-ac56-70564163d239\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " Jul 24 17:28:22.948575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948417 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rgdn\" (UniqueName: \"kubernetes.io/projected/7497ef0a-c1a0-4df2-ac56-70564163d239-kube-api-access-2rgdn\") pod \"7497ef0a-c1a0-4df2-ac56-70564163d239\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " Jul 24 17:28:22.948575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948436 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-console-config\") pod \"7497ef0a-c1a0-4df2-ac56-70564163d239\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " Jul 24 17:28:22.948575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948472 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c662871-978a-4f4c-89fa-959965d6b48c-extensions-socket-volume\") pod \"1c662871-978a-4f4c-89fa-959965d6b48c\" (UID: \"1c662871-978a-4f4c-89fa-959965d6b48c\") " Jul 24 17:28:22.948575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948493 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xrmg\" (UniqueName: \"kubernetes.io/projected/ab680537-ed21-4677-9889-5a6a578a20b3-kube-api-access-5xrmg\") pod \"ab680537-ed21-4677-9889-5a6a578a20b3\" (UID: \"ab680537-ed21-4677-9889-5a6a578a20b3\") " Jul 24 17:28:22.948575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948530 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-serving-cert\") pod \"7497ef0a-c1a0-4df2-ac56-70564163d239\" (UID: \"7497ef0a-c1a0-4df2-ac56-70564163d239\") " Jul 24 17:28:22.948575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948554 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cntfc\" (UniqueName: \"kubernetes.io/projected/1c662871-978a-4f4c-89fa-959965d6b48c-kube-api-access-cntfc\") pod \"1c662871-978a-4f4c-89fa-959965d6b48c\" (UID: \"1c662871-978a-4f4c-89fa-959965d6b48c\") " Jul 24 17:28:22.949015 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.948869 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7497ef0a-c1a0-4df2-ac56-70564163d239" (UID: "7497ef0a-c1a0-4df2-ac56-70564163d239"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:28:22.949079 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.949025 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7497ef0a-c1a0-4df2-ac56-70564163d239" (UID: "7497ef0a-c1a0-4df2-ac56-70564163d239"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:28:22.949137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.949072 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-console-config" (OuterVolumeSpecName: "console-config") pod "7497ef0a-c1a0-4df2-ac56-70564163d239" (UID: "7497ef0a-c1a0-4df2-ac56-70564163d239"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:28:22.949662 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.949606 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c662871-978a-4f4c-89fa-959965d6b48c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1c662871-978a-4f4c-89fa-959965d6b48c" (UID: "1c662871-978a-4f4c-89fa-959965d6b48c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:28:22.949662 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.949654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-service-ca" (OuterVolumeSpecName: "service-ca") pod "7497ef0a-c1a0-4df2-ac56-70564163d239" (UID: "7497ef0a-c1a0-4df2-ac56-70564163d239"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:28:22.951197 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.951173 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7497ef0a-c1a0-4df2-ac56-70564163d239" (UID: "7497ef0a-c1a0-4df2-ac56-70564163d239"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:28:22.951503 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.951480 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c662871-978a-4f4c-89fa-959965d6b48c-kube-api-access-cntfc" (OuterVolumeSpecName: "kube-api-access-cntfc") pod "1c662871-978a-4f4c-89fa-959965d6b48c" (UID: "1c662871-978a-4f4c-89fa-959965d6b48c"). InnerVolumeSpecName "kube-api-access-cntfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:28:22.951560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.951523 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7497ef0a-c1a0-4df2-ac56-70564163d239" (UID: "7497ef0a-c1a0-4df2-ac56-70564163d239"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:28:22.951560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.951535 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab680537-ed21-4677-9889-5a6a578a20b3-kube-api-access-5xrmg" (OuterVolumeSpecName: "kube-api-access-5xrmg") pod "ab680537-ed21-4677-9889-5a6a578a20b3" (UID: "ab680537-ed21-4677-9889-5a6a578a20b3"). InnerVolumeSpecName "kube-api-access-5xrmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:28:22.951634 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:22.951611 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7497ef0a-c1a0-4df2-ac56-70564163d239-kube-api-access-2rgdn" (OuterVolumeSpecName: "kube-api-access-2rgdn") pod "7497ef0a-c1a0-4df2-ac56-70564163d239" (UID: "7497ef0a-c1a0-4df2-ac56-70564163d239"). InnerVolumeSpecName "kube-api-access-2rgdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:28:23.050150 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050101 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-serving-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.050150 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050145 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cntfc\" (UniqueName: \"kubernetes.io/projected/1c662871-978a-4f4c-89fa-959965d6b48c-kube-api-access-cntfc\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.050150 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050155 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7497ef0a-c1a0-4df2-ac56-70564163d239-console-oauth-config\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.050356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050166 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.050356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050174 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-trusted-ca-bundle\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.050356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050186 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-oauth-serving-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.050356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050199 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgdn\" (UniqueName: \"kubernetes.io/projected/7497ef0a-c1a0-4df2-ac56-70564163d239-kube-api-access-2rgdn\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.050356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050207 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7497ef0a-c1a0-4df2-ac56-70564163d239-console-config\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.050356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050215 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c662871-978a-4f4c-89fa-959965d6b48c-extensions-socket-volume\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.050356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.050223 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xrmg\" (UniqueName: \"kubernetes.io/projected/ab680537-ed21-4677-9889-5a6a578a20b3-kube-api-access-5xrmg\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:23.643509 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.643483 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:23.643509 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.643508 2574 scope.go:117] "RemoveContainer" containerID="97fd585518e429e19ac99069cc860b9a4b9cc43e67f67006c6d0b147f50f8fd4" Jul 24 17:28:23.645506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.645479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" event={"ID":"c76563d8-aafb-446e-aae1-2fccbf2d4939","Type":"ContainerStarted","Data":"e6a5f737aa21597d85f1b889307596260dac9b9c235040e8b412f8648bc282a2"} Jul 24 17:28:23.645624 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.645532 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" event={"ID":"c76563d8-aafb-446e-aae1-2fccbf2d4939","Type":"ContainerStarted","Data":"a11464812ca5ff7951cadb7a1663f9fc7771d6b569043a09dc360f3eaf2885ac"} Jul 24 17:28:23.645624 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.645576 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:23.646349 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:23.646326 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:23.647166 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.647135 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654c7856b8-w5kfn" Jul 24 17:28:23.647166 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.647150 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654c7856b8-w5kfn" event={"ID":"7497ef0a-c1a0-4df2-ac56-70564163d239","Type":"ContainerDied","Data":"deb6561d63fcde8ab861814d3529e3d808db8c9ab617160573fea6c597405383"} Jul 24 17:28:23.651994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.648714 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:23.651994 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:23.648802 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h78z2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:23.651994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.650388 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" event={"ID":"c118badb-e8bc-4a8d-95ea-ed1e95b970dc","Type":"ContainerStarted","Data":"b86be310ee8409abe1246027d906a6d51c8e29dd42186f5b83c5517401fdb4cb"} Jul 24 17:28:23.653161 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.652277 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" event={"ID":"2e9cb16f-80db-48b7-8953-61ff7fa9189f","Type":"ContainerStarted","Data":"c3f8ccc161628e59f6a4deadf1b1972d8f42d1c0bb669c7d292b45adca6e2d59"} Jul 24 17:28:23.653161 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.652323 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" event={"ID":"2e9cb16f-80db-48b7-8953-61ff7fa9189f","Type":"ContainerStarted","Data":"6a2084afc209e36f26b6deabdf9bf9637a2ca2f8ad9abb2fc72a008a06e9a163"} Jul 24 17:28:23.653161 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.652414 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" Jul 24 17:28:23.654797 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.654773 2574 scope.go:117] "RemoveContainer" containerID="9b939532a87dc0f8b8f6e837e5b8a696711945e573fc8cbb3631b43305543532" Jul 24 17:28:23.655272 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:23.655237 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h78z2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:23.664354 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.664341 2574 scope.go:117] "RemoveContainer" containerID="8139c197a4ee2bf5d8405891e5f0b318c21ad31fe9567083053ac9e595af121a" Jul 24 17:28:23.674784 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:23.674761 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:23.674970 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.674936 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" podStartSLOduration=12.674924382 podStartE2EDuration="12.674924382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:23.672548463 +0000 UTC m=+660.338123101" watchObservedRunningTime="2026-07-24 17:28:23.674924382 +0000 UTC m=+660.340499018" Jul 24 17:28:23.692288 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.692265 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-654c7856b8-w5kfn"] Jul 24 17:28:23.693959 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:23.693934 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h78z2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:23.694313 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.694296 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-654c7856b8-w5kfn"] Jul 24 17:28:23.696156 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:23.696136 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:23.713255 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.713218 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jss8q" podStartSLOduration=2.100550889 podStartE2EDuration="22.713206643s" totalImagesPullingTime="20.612655754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:23.711991024 +0000 UTC m=+660.377565662" watchObservedRunningTime="2026-07-24 17:28:23.713206643 +0000 UTC m=+660.378781280" Jul 24 17:28:23.732511 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.732456 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" podStartSLOduration=12.732443219 podStartE2EDuration="12.732443219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:23.73092237 +0000 UTC m=+660.396497010" watchObservedRunningTime="2026-07-24 17:28:23.732443219 +0000 UTC m=+660.398017855" Jul 24 17:28:23.800278 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:23.800250 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xz7j2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xz7j2" Jul 24 17:28:23.801234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.801214 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c662871-978a-4f4c-89fa-959965d6b48c" path="/var/lib/kubelet/pods/1c662871-978a-4f4c-89fa-959965d6b48c/volumes" Jul 24 17:28:23.801577 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.801565 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7497ef0a-c1a0-4df2-ac56-70564163d239" path="/var/lib/kubelet/pods/7497ef0a-c1a0-4df2-ac56-70564163d239/volumes" Jul 24 17:28:23.801863 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:23.801851 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" path="/var/lib/kubelet/pods/ab680537-ed21-4677-9889-5a6a578a20b3/volumes" Jul 24 17:28:23.802718 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:28:23.802698 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h78z2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ab680537-ed21-4677-9889-5a6a578a20b3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h78z2" Jul 24 17:28:27.426594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.426560 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj"] Jul 24 17:28:27.427082 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.426750 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" podUID="c76563d8-aafb-446e-aae1-2fccbf2d4939" containerName="manager" containerID="cri-o://e6a5f737aa21597d85f1b889307596260dac9b9c235040e8b412f8648bc282a2" gracePeriod=10 Jul 24 17:28:27.428814 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.428791 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:27.675008 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.674977 2574 generic.go:356] "Generic (PLEG): container finished" podID="c76563d8-aafb-446e-aae1-2fccbf2d4939" containerID="e6a5f737aa21597d85f1b889307596260dac9b9c235040e8b412f8648bc282a2" exitCode=0 Jul 24 17:28:27.675130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.675050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" event={"ID":"c76563d8-aafb-446e-aae1-2fccbf2d4939","Type":"ContainerDied","Data":"e6a5f737aa21597d85f1b889307596260dac9b9c235040e8b412f8648bc282a2"} Jul 24 17:28:27.675130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.675096 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" event={"ID":"c76563d8-aafb-446e-aae1-2fccbf2d4939","Type":"ContainerDied","Data":"a11464812ca5ff7951cadb7a1663f9fc7771d6b569043a09dc360f3eaf2885ac"} Jul 24 17:28:27.675130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.675113 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a11464812ca5ff7951cadb7a1663f9fc7771d6b569043a09dc360f3eaf2885ac" Jul 24 17:28:27.681809 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.681758 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:27.786093 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.786061 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c76563d8-aafb-446e-aae1-2fccbf2d4939-extensions-socket-volume\") pod \"c76563d8-aafb-446e-aae1-2fccbf2d4939\" (UID: \"c76563d8-aafb-446e-aae1-2fccbf2d4939\") " Jul 24 17:28:27.786288 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.786183 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgnjc\" (UniqueName: \"kubernetes.io/projected/c76563d8-aafb-446e-aae1-2fccbf2d4939-kube-api-access-wgnjc\") pod \"c76563d8-aafb-446e-aae1-2fccbf2d4939\" (UID: \"c76563d8-aafb-446e-aae1-2fccbf2d4939\") " Jul 24 17:28:27.786421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.786399 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76563d8-aafb-446e-aae1-2fccbf2d4939-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c76563d8-aafb-446e-aae1-2fccbf2d4939" (UID: "c76563d8-aafb-446e-aae1-2fccbf2d4939"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:28:27.788318 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.788299 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76563d8-aafb-446e-aae1-2fccbf2d4939-kube-api-access-wgnjc" (OuterVolumeSpecName: "kube-api-access-wgnjc") pod "c76563d8-aafb-446e-aae1-2fccbf2d4939" (UID: "c76563d8-aafb-446e-aae1-2fccbf2d4939"). InnerVolumeSpecName "kube-api-access-wgnjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:28:27.886968 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.886927 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgnjc\" (UniqueName: \"kubernetes.io/projected/c76563d8-aafb-446e-aae1-2fccbf2d4939-kube-api-access-wgnjc\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:27.886968 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:27.886963 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c76563d8-aafb-446e-aae1-2fccbf2d4939-extensions-socket-volume\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:28:28.680371 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:28.680332 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj" Jul 24 17:28:28.704034 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:28.704006 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj"] Jul 24 17:28:28.710190 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:28.710170 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkbcj"] Jul 24 17:28:29.801345 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:29.801315 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76563d8-aafb-446e-aae1-2fccbf2d4939" path="/var/lib/kubelet/pods/c76563d8-aafb-446e-aae1-2fccbf2d4939/volumes" Jul 24 17:28:34.661147 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:34.661117 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gjkz8" Jul 24 17:28:43.827661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:43.827631 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb"] Jul 24 17:28:43.828140 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:43.828003 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7497ef0a-c1a0-4df2-ac56-70564163d239" containerName="console" Jul 24 17:28:43.828140 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:43.828015 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7497ef0a-c1a0-4df2-ac56-70564163d239" containerName="console" Jul 24 17:28:43.828140 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:43.828024 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76563d8-aafb-446e-aae1-2fccbf2d4939" containerName="manager" Jul 24 17:28:43.828140 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:43.828030 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76563d8-aafb-446e-aae1-2fccbf2d4939" containerName="manager" Jul 24 17:28:43.828140 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:43.828086 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7497ef0a-c1a0-4df2-ac56-70564163d239" containerName="console" Jul 24 17:28:43.828140 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:43.828095 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76563d8-aafb-446e-aae1-2fccbf2d4939" containerName="manager" Jul 24 17:28:44.033564 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.033528 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb"] Jul 24 17:28:44.033752 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.033650 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.036551 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.036527 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 17:28:44.036669 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.036550 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f844p\"" Jul 24 17:28:44.134836 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.134760 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.134836 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.134793 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda82824-adb8-43b7-bb20-fa840945d314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.134836 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.134812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda82824-adb8-43b7-bb20-fa840945d314-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.135039 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.134843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5j84\" (UniqueName: \"kubernetes.io/projected/cda82824-adb8-43b7-bb20-fa840945d314-kube-api-access-g5j84\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.135039 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.134888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda82824-adb8-43b7-bb20-fa840945d314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.135039 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.134924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.135039 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.134967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.135039 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.135037 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.135198 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.135068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.235801 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.235774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.235990 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.235812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.235990 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.235845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.235990 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.235861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda82824-adb8-43b7-bb20-fa840945d314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.235990 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.235970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda82824-adb8-43b7-bb20-fa840945d314-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.236201 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.235999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5j84\" (UniqueName: \"kubernetes.io/projected/cda82824-adb8-43b7-bb20-fa840945d314-kube-api-access-g5j84\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.236201 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.236027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda82824-adb8-43b7-bb20-fa840945d314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.236201 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.236060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.236201 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.236111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.236404 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.236206 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.236459 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.236406 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.236519 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.236464 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.236519 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.236502 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.236825 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.236681 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda82824-adb8-43b7-bb20-fa840945d314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.238300 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.238280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda82824-adb8-43b7-bb20-fa840945d314-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.238624 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.238605 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda82824-adb8-43b7-bb20-fa840945d314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.245031 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.245008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda82824-adb8-43b7-bb20-fa840945d314-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.245123 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.245087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5j84\" (UniqueName: \"kubernetes.io/projected/cda82824-adb8-43b7-bb20-fa840945d314-kube-api-access-g5j84\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rljcb\" (UID: \"cda82824-adb8-43b7-bb20-fa840945d314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.343717 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.343695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:44.509411 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.509374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb"] Jul 24 17:28:44.510940 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:28:44.510908 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda82824_adb8_43b7_bb20_fa840945d314.slice/crio-57af4f845917c1a9c8f03595c6ed2276bf4b45adaff0fc5d79dad89fcb9d763e WatchSource:0}: Error finding container 57af4f845917c1a9c8f03595c6ed2276bf4b45adaff0fc5d79dad89fcb9d763e: Status 404 returned error can't find the container with id 57af4f845917c1a9c8f03595c6ed2276bf4b45adaff0fc5d79dad89fcb9d763e Jul 24 17:28:44.764773 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:44.764744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" event={"ID":"cda82824-adb8-43b7-bb20-fa840945d314","Type":"ContainerStarted","Data":"57af4f845917c1a9c8f03595c6ed2276bf4b45adaff0fc5d79dad89fcb9d763e"} Jul 24 17:28:47.933646 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:47.933608 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:28:47.934022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:47.933697 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:28:47.934022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:47.933769 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:28:48.792858 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:48.792820 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" event={"ID":"cda82824-adb8-43b7-bb20-fa840945d314","Type":"ContainerStarted","Data":"b29b3258c079dbf02d75e70cf3d18ce22809f5b3173cea75811ea9f6d7b75bd4"} Jul 24 17:28:48.823223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:48.823166 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" podStartSLOduration=2.402714991 podStartE2EDuration="5.82315064s" totalImagesPullingTime="3.420435649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:48.82067146 +0000 UTC m=+685.486246095" watchObservedRunningTime="2026-07-24 17:28:48.82315064 +0000 UTC m=+685.488725277" Jul 24 17:28:49.344589 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:49.344551 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:49.349137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:49.349115 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:49.802031 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:49.802001 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:49.802213 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:49.802070 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:28:49.917828 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:49.917792 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-grfvq"] Jul 24 17:28:49.995602 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:49.995565 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-grfvq"] Jul 24 17:28:49.995788 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:49.995693 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:28:49.998526 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:49.998502 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 17:28:50.008017 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.007992 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-grfvq"] Jul 24 17:28:50.188997 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.188916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64xf\" (UniqueName: \"kubernetes.io/projected/c15fe981-d374-4538-b694-c7bbd434c215-kube-api-access-j64xf\") pod \"limitador-limitador-64c94fd64f-grfvq\" (UID: \"c15fe981-d374-4538-b694-c7bbd434c215\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:28:50.188997 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.188955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c15fe981-d374-4538-b694-c7bbd434c215-config-file\") pod \"limitador-limitador-64c94fd64f-grfvq\" (UID: \"c15fe981-d374-4538-b694-c7bbd434c215\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:28:50.279101 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.279059 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:28:50.283055 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.283036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:28:50.289543 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.289517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64xf\" (UniqueName: \"kubernetes.io/projected/c15fe981-d374-4538-b694-c7bbd434c215-kube-api-access-j64xf\") pod \"limitador-limitador-64c94fd64f-grfvq\" (UID: \"c15fe981-d374-4538-b694-c7bbd434c215\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:28:50.289649 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.289558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c15fe981-d374-4538-b694-c7bbd434c215-config-file\") pod \"limitador-limitador-64c94fd64f-grfvq\" (UID: \"c15fe981-d374-4538-b694-c7bbd434c215\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:28:50.292245 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.290711 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c15fe981-d374-4538-b694-c7bbd434c215-config-file\") pod \"limitador-limitador-64c94fd64f-grfvq\" (UID: \"c15fe981-d374-4538-b694-c7bbd434c215\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:28:50.292245 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.291850 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:28:50.303615 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.303581 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64xf\" (UniqueName: \"kubernetes.io/projected/c15fe981-d374-4538-b694-c7bbd434c215-kube-api-access-j64xf\") pod \"limitador-limitador-64c94fd64f-grfvq\" (UID: \"c15fe981-d374-4538-b694-c7bbd434c215\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:28:50.308447 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.308422 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:28:50.321315 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.321296 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:28:50.390128 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.390101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1d3c6b0-62fa-4107-9912-db946a0ebe59-config-file\") pod \"limitador-limitador-54d6d988d5-9rth9\" (UID: \"e1d3c6b0-62fa-4107-9912-db946a0ebe59\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:28:50.390475 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.390133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ggnk\" (UniqueName: \"kubernetes.io/projected/e1d3c6b0-62fa-4107-9912-db946a0ebe59-kube-api-access-2ggnk\") pod \"limitador-limitador-54d6d988d5-9rth9\" (UID: \"e1d3c6b0-62fa-4107-9912-db946a0ebe59\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:28:50.445056 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.445028 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-grfvq"] Jul 24 17:28:50.445904 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:28:50.445876 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15fe981_d374_4538_b694_c7bbd434c215.slice/crio-8a19f6453ebb1e12e4163672583e0fd9150e16efcae7084f978da415c7c7b842 WatchSource:0}: Error finding container 8a19f6453ebb1e12e4163672583e0fd9150e16efcae7084f978da415c7c7b842: Status 404 returned error can't find the container with id 8a19f6453ebb1e12e4163672583e0fd9150e16efcae7084f978da415c7c7b842 Jul 24 17:28:50.491197 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.491163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1d3c6b0-62fa-4107-9912-db946a0ebe59-config-file\") pod \"limitador-limitador-54d6d988d5-9rth9\" (UID: \"e1d3c6b0-62fa-4107-9912-db946a0ebe59\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:28:50.491197 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.491197 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ggnk\" (UniqueName: \"kubernetes.io/projected/e1d3c6b0-62fa-4107-9912-db946a0ebe59-kube-api-access-2ggnk\") pod \"limitador-limitador-54d6d988d5-9rth9\" (UID: \"e1d3c6b0-62fa-4107-9912-db946a0ebe59\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:28:50.491747 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.491711 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1d3c6b0-62fa-4107-9912-db946a0ebe59-config-file\") pod \"limitador-limitador-54d6d988d5-9rth9\" (UID: \"e1d3c6b0-62fa-4107-9912-db946a0ebe59\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:28:50.500833 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.500807 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ggnk\" (UniqueName: \"kubernetes.io/projected/e1d3c6b0-62fa-4107-9912-db946a0ebe59-kube-api-access-2ggnk\") pod \"limitador-limitador-54d6d988d5-9rth9\" (UID: \"e1d3c6b0-62fa-4107-9912-db946a0ebe59\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:28:50.596445 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.596413 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:28:50.741779 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.741755 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:28:50.743967 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:28:50.743939 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d3c6b0_62fa_4107_9912_db946a0ebe59.slice/crio-4018f13acb6eec63befb9bed749ac68dcdcce6def421c505a8a6e8748e9dd1b5 WatchSource:0}: Error finding container 4018f13acb6eec63befb9bed749ac68dcdcce6def421c505a8a6e8748e9dd1b5: Status 404 returned error can't find the container with id 4018f13acb6eec63befb9bed749ac68dcdcce6def421c505a8a6e8748e9dd1b5 Jul 24 17:28:50.805536 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.805508 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" event={"ID":"c15fe981-d374-4538-b694-c7bbd434c215","Type":"ContainerStarted","Data":"8a19f6453ebb1e12e4163672583e0fd9150e16efcae7084f978da415c7c7b842"} Jul 24 17:28:50.806630 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:50.806605 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" event={"ID":"e1d3c6b0-62fa-4107-9912-db946a0ebe59","Type":"ContainerStarted","Data":"4018f13acb6eec63befb9bed749ac68dcdcce6def421c505a8a6e8748e9dd1b5"} Jul 24 17:28:53.829846 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:53.829809 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" event={"ID":"c15fe981-d374-4538-b694-c7bbd434c215","Type":"ContainerStarted","Data":"ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887"} Jul 24 17:28:53.830324 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:53.830011 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:28:53.831418 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:53.831395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" event={"ID":"e1d3c6b0-62fa-4107-9912-db946a0ebe59","Type":"ContainerStarted","Data":"c9ce9b3816e87f3fc3c37365989b890c3db2344a81cb8143b3f1094d35f7c394"} Jul 24 17:28:53.831534 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:53.831459 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:28:53.848703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:53.848659 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" podStartSLOduration=2.062938066 podStartE2EDuration="4.848646252s" totalImagesPullingTime="2.785708186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:53.846393558 +0000 UTC m=+690.511968192" watchObservedRunningTime="2026-07-24 17:28:53.848646252 +0000 UTC m=+690.514220888" Jul 24 17:28:53.864279 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:28:53.864236 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" podStartSLOduration=1.369301259 podStartE2EDuration="3.864223111s" totalImagesPullingTime="2.494921852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:28:53.862105412 +0000 UTC m=+690.527680050" watchObservedRunningTime="2026-07-24 17:28:53.864223111 +0000 UTC m=+690.529797782" Jul 24 17:29:04.839274 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:04.839247 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9rth9" Jul 24 17:29:04.839650 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:04.839316 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:29:04.905477 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:04.905442 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-grfvq"] Jul 24 17:29:04.905647 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:04.905630 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" podUID="c15fe981-d374-4538-b694-c7bbd434c215" containerName="limitador" containerID="cri-o://ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887" gracePeriod=30 Jul 24 17:29:05.858795 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.858765 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:29:05.899213 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.899178 2574 generic.go:356] "Generic (PLEG): container finished" podID="c15fe981-d374-4538-b694-c7bbd434c215" containerID="ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887" exitCode=0 Jul 24 17:29:05.899375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.899218 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" Jul 24 17:29:05.899375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.899262 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" event={"ID":"c15fe981-d374-4538-b694-c7bbd434c215","Type":"ContainerDied","Data":"ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887"} Jul 24 17:29:05.899375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.899327 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-grfvq" event={"ID":"c15fe981-d374-4538-b694-c7bbd434c215","Type":"ContainerDied","Data":"8a19f6453ebb1e12e4163672583e0fd9150e16efcae7084f978da415c7c7b842"} Jul 24 17:29:05.899375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.899357 2574 scope.go:117] "RemoveContainer" containerID="ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887" Jul 24 17:29:05.908383 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.908224 2574 scope.go:117] "RemoveContainer" containerID="ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887" Jul 24 17:29:05.908496 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:29:05.908478 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887\": container with ID starting with ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887 not found: ID does not exist" containerID="ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887" Jul 24 17:29:05.908535 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.908506 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887"} err="failed to get container status \"ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887\": rpc error: code = NotFound desc = could not find container \"ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887\": container with ID starting with ed8839dfa51beb66af12c751ebe882c54458c744e81bf694aa2f86c7b7be8887 not found: ID does not exist" Jul 24 17:29:05.921819 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.921802 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j64xf\" (UniqueName: \"kubernetes.io/projected/c15fe981-d374-4538-b694-c7bbd434c215-kube-api-access-j64xf\") pod \"c15fe981-d374-4538-b694-c7bbd434c215\" (UID: \"c15fe981-d374-4538-b694-c7bbd434c215\") " Jul 24 17:29:05.921893 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.921859 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c15fe981-d374-4538-b694-c7bbd434c215-config-file\") pod \"c15fe981-d374-4538-b694-c7bbd434c215\" (UID: \"c15fe981-d374-4538-b694-c7bbd434c215\") " Jul 24 17:29:05.922225 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.922203 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15fe981-d374-4538-b694-c7bbd434c215-config-file" (OuterVolumeSpecName: "config-file") pod "c15fe981-d374-4538-b694-c7bbd434c215" (UID: "c15fe981-d374-4538-b694-c7bbd434c215"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:29:05.924043 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:05.924020 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15fe981-d374-4538-b694-c7bbd434c215-kube-api-access-j64xf" (OuterVolumeSpecName: "kube-api-access-j64xf") pod "c15fe981-d374-4538-b694-c7bbd434c215" (UID: "c15fe981-d374-4538-b694-c7bbd434c215"). InnerVolumeSpecName "kube-api-access-j64xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:29:06.022966 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.022888 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j64xf\" (UniqueName: \"kubernetes.io/projected/c15fe981-d374-4538-b694-c7bbd434c215-kube-api-access-j64xf\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:29:06.022966 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.022915 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c15fe981-d374-4538-b694-c7bbd434c215-config-file\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:29:06.225781 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.225750 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-grfvq"] Jul 24 17:29:06.227838 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.227819 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-grfvq"] Jul 24 17:29:06.619301 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.619264 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs"] Jul 24 17:29:06.619652 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.619637 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15fe981-d374-4538-b694-c7bbd434c215" containerName="limitador" Jul 24 17:29:06.619741 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.619654 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15fe981-d374-4538-b694-c7bbd434c215" containerName="limitador" Jul 24 17:29:06.619789 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.619755 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15fe981-d374-4538-b694-c7bbd434c215" containerName="limitador" Jul 24 17:29:06.624110 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.624094 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:06.627303 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.627271 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 17:29:06.627439 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.627312 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 17:29:06.627439 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.627315 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 17:29:06.627439 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.627358 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5d8sg\"" Jul 24 17:29:06.635057 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.635024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs"] Jul 24 17:29:06.730895 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.730864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e79aff8c-138d-453f-9470-4036151fb467-data\") pod \"postgres-69fb4594cf-qlxrs\" (UID: \"e79aff8c-138d-453f-9470-4036151fb467\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:06.731077 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.730933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtmc\" (UniqueName: \"kubernetes.io/projected/e79aff8c-138d-453f-9470-4036151fb467-kube-api-access-2dtmc\") pod \"postgres-69fb4594cf-qlxrs\" (UID: \"e79aff8c-138d-453f-9470-4036151fb467\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:06.831871 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.831839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e79aff8c-138d-453f-9470-4036151fb467-data\") pod \"postgres-69fb4594cf-qlxrs\" (UID: \"e79aff8c-138d-453f-9470-4036151fb467\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:06.832042 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.831920 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtmc\" (UniqueName: \"kubernetes.io/projected/e79aff8c-138d-453f-9470-4036151fb467-kube-api-access-2dtmc\") pod \"postgres-69fb4594cf-qlxrs\" (UID: \"e79aff8c-138d-453f-9470-4036151fb467\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:06.832234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.832217 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e79aff8c-138d-453f-9470-4036151fb467-data\") pod \"postgres-69fb4594cf-qlxrs\" (UID: \"e79aff8c-138d-453f-9470-4036151fb467\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:06.840707 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.840680 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtmc\" (UniqueName: \"kubernetes.io/projected/e79aff8c-138d-453f-9470-4036151fb467-kube-api-access-2dtmc\") pod \"postgres-69fb4594cf-qlxrs\" (UID: \"e79aff8c-138d-453f-9470-4036151fb467\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:06.936136 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:06.936055 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:07.075738 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:07.075702 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs"] Jul 24 17:29:07.077145 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:29:07.077110 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79aff8c_138d_453f_9470_4036151fb467.slice/crio-54f7b6dce6a6a2f08f87131a5a2093a845f2327c8d054ebbc5c90d9bc78dea7d WatchSource:0}: Error finding container 54f7b6dce6a6a2f08f87131a5a2093a845f2327c8d054ebbc5c90d9bc78dea7d: Status 404 returned error can't find the container with id 54f7b6dce6a6a2f08f87131a5a2093a845f2327c8d054ebbc5c90d9bc78dea7d Jul 24 17:29:07.801411 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:07.801378 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c15fe981-d374-4538-b694-c7bbd434c215" path="/var/lib/kubelet/pods/c15fe981-d374-4538-b694-c7bbd434c215/volumes" Jul 24 17:29:07.912494 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:07.912461 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" event={"ID":"e79aff8c-138d-453f-9470-4036151fb467","Type":"ContainerStarted","Data":"54f7b6dce6a6a2f08f87131a5a2093a845f2327c8d054ebbc5c90d9bc78dea7d"} Jul 24 17:29:12.948483 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:12.948451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" event={"ID":"e79aff8c-138d-453f-9470-4036151fb467","Type":"ContainerStarted","Data":"2d48675522ab5e9a575fc688b4ed414c30b97a23d6297bc27a06ac071865f27e"} Jul 24 17:29:12.948947 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:12.948632 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:12.967067 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:12.967017 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" podStartSLOduration=1.74586511 podStartE2EDuration="6.966997583s" totalImagesPullingTime="5.221132473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:12.966957025 +0000 UTC m=+709.632531686" watchObservedRunningTime="2026-07-24 17:29:12.966997583 +0000 UTC m=+709.632572220" Jul 24 17:29:18.983809 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:18.983708 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qlxrs" Jul 24 17:29:19.824678 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:19.824643 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58b4dd9587-52xk4"] Jul 24 17:29:19.827291 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:19.827275 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:29:19.831382 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:19.831360 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 24 17:29:19.831480 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:19.831360 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4cwtj\"" Jul 24 17:29:19.835375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:19.835353 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58b4dd9587-52xk4"] Jul 24 17:29:19.952704 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:19.952676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltpv\" (UniqueName: \"kubernetes.io/projected/04fe3992-aad0-420a-92f4-3064493c6e73-kube-api-access-7ltpv\") pod \"authorino-58b4dd9587-52xk4\" (UID: \"04fe3992-aad0-420a-92f4-3064493c6e73\") " pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:29:19.952704 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:19.952707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/04fe3992-aad0-420a-92f4-3064493c6e73-tls-cert\") pod \"authorino-58b4dd9587-52xk4\" (UID: \"04fe3992-aad0-420a-92f4-3064493c6e73\") " pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:29:20.053373 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:20.053337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltpv\" (UniqueName: \"kubernetes.io/projected/04fe3992-aad0-420a-92f4-3064493c6e73-kube-api-access-7ltpv\") pod \"authorino-58b4dd9587-52xk4\" (UID: \"04fe3992-aad0-420a-92f4-3064493c6e73\") " pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:29:20.053373 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:20.053374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/04fe3992-aad0-420a-92f4-3064493c6e73-tls-cert\") pod \"authorino-58b4dd9587-52xk4\" (UID: \"04fe3992-aad0-420a-92f4-3064493c6e73\") " pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:29:20.055975 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:20.055952 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/04fe3992-aad0-420a-92f4-3064493c6e73-tls-cert\") pod \"authorino-58b4dd9587-52xk4\" (UID: \"04fe3992-aad0-420a-92f4-3064493c6e73\") " pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:29:20.061743 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:20.061704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltpv\" (UniqueName: \"kubernetes.io/projected/04fe3992-aad0-420a-92f4-3064493c6e73-kube-api-access-7ltpv\") pod \"authorino-58b4dd9587-52xk4\" (UID: \"04fe3992-aad0-420a-92f4-3064493c6e73\") " pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:29:20.136902 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:20.136818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:29:20.271106 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:20.271082 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58b4dd9587-52xk4"] Jul 24 17:29:20.272456 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:29:20.272431 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04fe3992_aad0_420a_92f4_3064493c6e73.slice/crio-3b7c65415d6c403d476c1719d754444e7f11c7f4322a4154ae6c50a33b7a595c WatchSource:0}: Error finding container 3b7c65415d6c403d476c1719d754444e7f11c7f4322a4154ae6c50a33b7a595c: Status 404 returned error can't find the container with id 3b7c65415d6c403d476c1719d754444e7f11c7f4322a4154ae6c50a33b7a595c Jul 24 17:29:20.997125 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:20.997064 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b4dd9587-52xk4" event={"ID":"04fe3992-aad0-420a-92f4-3064493c6e73","Type":"ContainerStarted","Data":"3b7c65415d6c403d476c1719d754444e7f11c7f4322a4154ae6c50a33b7a595c"} Jul 24 17:29:24.017039 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:24.016995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b4dd9587-52xk4" event={"ID":"04fe3992-aad0-420a-92f4-3064493c6e73","Type":"ContainerStarted","Data":"d25aa96af15f05eb95175d706441e2cdb0fff7ac1e11968b36651348b1088e49"} Jul 24 17:29:24.035883 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:29:24.035797 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58b4dd9587-52xk4" podStartSLOduration=1.8391757549999999 podStartE2EDuration="5.035783041s" totalImagesPullingTime="3.196607286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:24.033685307 +0000 UTC m=+720.699259944" watchObservedRunningTime="2026-07-24 17:29:24.035783041 +0000 UTC m=+720.701357678" Jul 24 17:30:05.069774 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:05.069733 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:30:13.297419 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:13.297387 2574 generic.go:356] "Generic (PLEG): container finished" podID="cda82824-adb8-43b7-bb20-fa840945d314" containerID="b29b3258c079dbf02d75e70cf3d18ce22809f5b3173cea75811ea9f6d7b75bd4" exitCode=0 Jul 24 17:30:13.297858 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:13.297457 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" event={"ID":"cda82824-adb8-43b7-bb20-fa840945d314","Type":"ContainerDied","Data":"b29b3258c079dbf02d75e70cf3d18ce22809f5b3173cea75811ea9f6d7b75bd4"} Jul 24 17:30:13.297909 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:13.297859 2574 scope.go:117] "RemoveContainer" containerID="b29b3258c079dbf02d75e70cf3d18ce22809f5b3173cea75811ea9f6d7b75bd4" Jul 24 17:30:13.298880 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:13.298850 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:30:13.298951 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:13.298923 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:30:13.298987 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:13.298956 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:30:14.305228 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:14.305198 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" event={"ID":"cda82824-adb8-43b7-bb20-fa840945d314","Type":"ContainerStarted","Data":"87c253766cd60d1ac7bc43cc00c5c11ec5e7aec8d89177f2ba6efb8c9a41d9cf"} Jul 24 17:30:14.345145 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:14.345106 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:30:14.345351 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:14.345265 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rljcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 24 17:30:14.345351 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:14.345314 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" podUID="cda82824-adb8-43b7-bb20-fa840945d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 24 17:30:15.345028 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:15.344981 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rljcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 24 17:30:15.345543 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:15.345053 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" podUID="cda82824-adb8-43b7-bb20-fa840945d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 24 17:30:16.344447 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:16.344401 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rljcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 24 17:30:16.344633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:16.344472 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" podUID="cda82824-adb8-43b7-bb20-fa840945d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 24 17:30:17.344626 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:17.344587 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rljcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 24 17:30:17.345298 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:17.345222 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" podUID="cda82824-adb8-43b7-bb20-fa840945d314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 24 17:30:18.348174 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:18.348146 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:30:18.348648 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:18.348534 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:30:18.349214 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:18.349196 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rljcb" Jul 24 17:30:22.863651 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:22.863612 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:30:32.891660 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.891624 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67"] Jul 24 17:30:32.895703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.895686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:32.899105 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.899084 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 17:30:32.900566 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.900544 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 17:30:32.900566 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.900560 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4mjtq\"" Jul 24 17:30:32.900817 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.900543 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 17:30:32.903843 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.903820 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67"] Jul 24 17:30:32.995550 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.995520 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:32.995758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.995555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a398cbf-46fe-4147-89cb-0c68b116f912-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:32.995758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.995595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:32.995758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.995667 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:32.995758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.995685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:32.995758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:32.995714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrls\" (UniqueName: \"kubernetes.io/projected/9a398cbf-46fe-4147-89cb-0c68b116f912-kube-api-access-rxrls\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.096416 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.096387 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.096416 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.096421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.096663 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.096444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrls\" (UniqueName: \"kubernetes.io/projected/9a398cbf-46fe-4147-89cb-0c68b116f912-kube-api-access-rxrls\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.096663 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.096473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.096663 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.096490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a398cbf-46fe-4147-89cb-0c68b116f912-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.096663 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.096529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.096941 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.096915 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.096999 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.096919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.096999 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.096965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.098836 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.098812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a398cbf-46fe-4147-89cb-0c68b116f912-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.099106 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.099090 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a398cbf-46fe-4147-89cb-0c68b116f912-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.108029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.107970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrls\" (UniqueName: \"kubernetes.io/projected/9a398cbf-46fe-4147-89cb-0c68b116f912-kube-api-access-rxrls\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8zs67\" (UID: \"9a398cbf-46fe-4147-89cb-0c68b116f912\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.207278 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.207245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:33.344784 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.344758 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67"] Jul 24 17:30:33.346946 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:30:33.346920 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a398cbf_46fe_4147_89cb_0c68b116f912.slice/crio-230b5829065a3fc32e766c6b0aef99378c06a05b5e109cad62c17cc2edfb8649 WatchSource:0}: Error finding container 230b5829065a3fc32e766c6b0aef99378c06a05b5e109cad62c17cc2edfb8649: Status 404 returned error can't find the container with id 230b5829065a3fc32e766c6b0aef99378c06a05b5e109cad62c17cc2edfb8649 Jul 24 17:30:33.412072 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.412038 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" event={"ID":"9a398cbf-46fe-4147-89cb-0c68b116f912","Type":"ContainerStarted","Data":"230b5829065a3fc32e766c6b0aef99378c06a05b5e109cad62c17cc2edfb8649"} Jul 24 17:30:33.666873 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:33.666785 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:30:35.424943 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:35.424906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" event={"ID":"9a398cbf-46fe-4147-89cb-0c68b116f912","Type":"ContainerStarted","Data":"374f0e0d6c562089703d163074d09e20947dcb02d889f2f60ae8571b75ba4fb5"} Jul 24 17:30:35.444089 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:35.444038 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" podStartSLOduration=2.318701055 podStartE2EDuration="3.444021878s" totalImagesPullingTime="1.125320823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:30:35.442086462 +0000 UTC m=+792.107661101" watchObservedRunningTime="2026-07-24 17:30:35.444021878 +0000 UTC m=+792.109596508" Jul 24 17:30:37.157243 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:37.157206 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:30:42.868113 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:42.868079 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:30:43.207668 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:43.207633 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:43.207928 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:43.207786 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:43.221061 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:43.221034 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:30:43.485518 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:30:43.485444 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8zs67" Jul 24 17:31:02.469334 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:31:02.469296 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:31:15.678103 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:31:15.678052 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:32:11.782349 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:11.782311 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58b4dd9587-52xk4"] Jul 24 17:32:11.782836 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:11.782502 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58b4dd9587-52xk4" podUID="04fe3992-aad0-420a-92f4-3064493c6e73" containerName="authorino" containerID="cri-o://d25aa96af15f05eb95175d706441e2cdb0fff7ac1e11968b36651348b1088e49" gracePeriod=30 Jul 24 17:32:11.969482 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:11.969451 2574 generic.go:356] "Generic (PLEG): container finished" podID="04fe3992-aad0-420a-92f4-3064493c6e73" containerID="d25aa96af15f05eb95175d706441e2cdb0fff7ac1e11968b36651348b1088e49" exitCode=0 Jul 24 17:32:11.969692 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:11.969505 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b4dd9587-52xk4" event={"ID":"04fe3992-aad0-420a-92f4-3064493c6e73","Type":"ContainerDied","Data":"d25aa96af15f05eb95175d706441e2cdb0fff7ac1e11968b36651348b1088e49"} Jul 24 17:32:12.038282 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.038216 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:32:12.087539 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.087500 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ltpv\" (UniqueName: \"kubernetes.io/projected/04fe3992-aad0-420a-92f4-3064493c6e73-kube-api-access-7ltpv\") pod \"04fe3992-aad0-420a-92f4-3064493c6e73\" (UID: \"04fe3992-aad0-420a-92f4-3064493c6e73\") " Jul 24 17:32:12.087697 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.087627 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/04fe3992-aad0-420a-92f4-3064493c6e73-tls-cert\") pod \"04fe3992-aad0-420a-92f4-3064493c6e73\" (UID: \"04fe3992-aad0-420a-92f4-3064493c6e73\") " Jul 24 17:32:12.089688 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.089657 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fe3992-aad0-420a-92f4-3064493c6e73-kube-api-access-7ltpv" (OuterVolumeSpecName: "kube-api-access-7ltpv") pod "04fe3992-aad0-420a-92f4-3064493c6e73" (UID: "04fe3992-aad0-420a-92f4-3064493c6e73"). InnerVolumeSpecName "kube-api-access-7ltpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:32:12.099760 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.099717 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fe3992-aad0-420a-92f4-3064493c6e73-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "04fe3992-aad0-420a-92f4-3064493c6e73" (UID: "04fe3992-aad0-420a-92f4-3064493c6e73"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:32:12.189151 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.189120 2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/04fe3992-aad0-420a-92f4-3064493c6e73-tls-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:32:12.189151 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.189145 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ltpv\" (UniqueName: \"kubernetes.io/projected/04fe3992-aad0-420a-92f4-3064493c6e73-kube-api-access-7ltpv\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:32:12.976604 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.976573 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b4dd9587-52xk4" Jul 24 17:32:12.977026 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.976622 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b4dd9587-52xk4" event={"ID":"04fe3992-aad0-420a-92f4-3064493c6e73","Type":"ContainerDied","Data":"3b7c65415d6c403d476c1719d754444e7f11c7f4322a4154ae6c50a33b7a595c"} Jul 24 17:32:12.977026 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:12.976676 2574 scope.go:117] "RemoveContainer" containerID="d25aa96af15f05eb95175d706441e2cdb0fff7ac1e11968b36651348b1088e49" Jul 24 17:32:13.048517 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:13.046806 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58b4dd9587-52xk4"] Jul 24 17:32:13.055630 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:13.055602 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58b4dd9587-52xk4"] Jul 24 17:32:13.801566 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:13.801533 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04fe3992-aad0-420a-92f4-3064493c6e73" path="/var/lib/kubelet/pods/04fe3992-aad0-420a-92f4-3064493c6e73/volumes" Jul 24 17:32:23.767590 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:23.767562 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:32:23.769249 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:23.769228 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:32:41.685020 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:41.684984 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:32:51.280140 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:32:51.280104 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:33:03.200709 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:33:03.200673 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:34:23.820385 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:34:23.820357 2574 scope.go:117] "RemoveContainer" containerID="e6a5f737aa21597d85f1b889307596260dac9b9c235040e8b412f8648bc282a2" Jul 24 17:34:47.874657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:34:47.874620 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:34:59.064156 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:34:59.064106 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:35:34.761660 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:35:34.761583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:35:43.864808 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:35:43.864774 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:35:52.669084 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:35:52.669048 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:36:04.163502 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:36:04.163466 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:36:14.173072 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:36:14.173033 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:36:26.375082 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:36:26.375049 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:36:36.762563 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:36:36.762526 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:36:46.869477 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:36:46.869441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:37:08.576970 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:37:08.576934 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:37:18.276053 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:37:18.276022 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:37:23.804247 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:37:23.804222 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:37:23.807611 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:37:23.807589 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:37:42.362427 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:37:42.362388 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:37:52.668962 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:37:52.668918 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:38:07.880790 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:38:07.880754 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:38:18.281824 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:38:18.281736 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:38:28.767978 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:38:28.767944 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:38:38.567647 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:38:38.567601 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:38:47.680291 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:38:47.680252 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:40:37.270011 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:40:37.269970 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:40:46.763448 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:40:46.763415 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:40:59.369375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:40:59.369338 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:41:11.076745 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:41:11.076685 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:41:22.063854 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:41:22.063778 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:41:32.465188 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:41:32.465154 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:41:42.973344 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:41:42.973304 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:41:52.464304 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:41:52.464253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:42:04.369109 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:42:04.369070 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:42:13.062826 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:42:13.062788 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:42:23.767417 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:42:23.767386 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:42:23.838431 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:42:23.838408 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:42:23.844661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:42:23.844639 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:42:45.469577 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:42:45.469482 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:42:54.864509 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:42:54.864465 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:43:07.265379 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:43:07.265337 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:43:21.063696 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:43:21.063649 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:43:44.670885 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:43:44.670850 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:43:58.523007 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:43:58.522971 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:44:05.868617 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:44:05.868583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:44:15.377556 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:44:15.377514 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:44:23.565072 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:44:23.565041 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:44:32.860549 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:44:32.860512 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:44:42.468976 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:44:42.468942 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:44:53.076445 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:44:53.076411 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:45:00.150882 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.150850 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89"] Jul 24 17:45:00.151315 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.151240 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fe3992-aad0-420a-92f4-3064493c6e73" containerName="authorino" Jul 24 17:45:00.151315 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.151251 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fe3992-aad0-420a-92f4-3064493c6e73" containerName="authorino" Jul 24 17:45:00.151315 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.151311 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="04fe3992-aad0-420a-92f4-3064493c6e73" containerName="authorino" Jul 24 17:45:00.153334 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.153317 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89" Jul 24 17:45:00.156442 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.156426 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:45:00.162433 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.162407 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89"] Jul 24 17:45:00.164609 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.164586 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89" Jul 24 17:45:00.299401 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.299379 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89"] Jul 24 17:45:00.301406 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:45:00.301380 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f8f502_62de_407d_9d45_84f7d2c192e0.slice/crio-2f1c44295c351a3bc34d3090c7475e347f11608f95e541f7f347355f13a2b22d WatchSource:0}: Error finding container 2f1c44295c351a3bc34d3090c7475e347f11608f95e541f7f347355f13a2b22d: Status 404 returned error can't find the container with id 2f1c44295c351a3bc34d3090c7475e347f11608f95e541f7f347355f13a2b22d Jul 24 17:45:00.303434 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:00.303404 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:45:01.184197 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:01.184163 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89" event={"ID":"c1f8f502-62de-407d-9d45-84f7d2c192e0","Type":"ContainerStarted","Data":"2f1c44295c351a3bc34d3090c7475e347f11608f95e541f7f347355f13a2b22d"} Jul 24 17:45:02.074427 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:02.074388 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:45:03.200031 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:03.199991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89" event={"ID":"c1f8f502-62de-407d-9d45-84f7d2c192e0","Type":"ContainerStarted","Data":"7c916866ef6f0c460d29c27cee45ce06e802a651ceeaaa5545389f724cc2fbee"} Jul 24 17:45:04.206228 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:04.206195 2574 generic.go:356] "Generic (PLEG): container finished" podID="c1f8f502-62de-407d-9d45-84f7d2c192e0" containerID="7c916866ef6f0c460d29c27cee45ce06e802a651ceeaaa5545389f724cc2fbee" exitCode=0 Jul 24 17:45:04.206611 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:04.206249 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89" event={"ID":"c1f8f502-62de-407d-9d45-84f7d2c192e0","Type":"ContainerDied","Data":"7c916866ef6f0c460d29c27cee45ce06e802a651ceeaaa5545389f724cc2fbee"} Jul 24 17:45:05.350908 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:05.350881 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89" Jul 24 17:45:06.218998 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:06.218967 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89" Jul 24 17:45:06.218998 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:06.218989 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-n9r89" event={"ID":"c1f8f502-62de-407d-9d45-84f7d2c192e0","Type":"ContainerDied","Data":"2f1c44295c351a3bc34d3090c7475e347f11608f95e541f7f347355f13a2b22d"} Jul 24 17:45:06.219210 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:06.219056 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f1c44295c351a3bc34d3090c7475e347f11608f95e541f7f347355f13a2b22d" Jul 24 17:45:11.365421 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:11.365386 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:45:24.464917 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:24.464881 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:45:33.713254 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:33.713219 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:45:43.476353 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:43.476321 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:45:53.986920 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:45:53.986889 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:46:04.566357 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:46:04.566317 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:46:11.362297 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:46:11.362254 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:46:25.767703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:46:25.767666 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:46:37.473189 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:46:37.473156 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:46:46.779791 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:46:46.779749 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:47:00.079601 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:00.079565 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:47:10.280778 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:10.280741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:47:19.582285 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:19.582190 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:47:21.319913 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.319884 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j"] Jul 24 17:47:21.320402 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.320385 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f8f502-62de-407d-9d45-84f7d2c192e0" containerName="cleanup" Jul 24 17:47:21.320445 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.320405 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f8f502-62de-407d-9d45-84f7d2c192e0" containerName="cleanup" Jul 24 17:47:21.320497 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.320488 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f8f502-62de-407d-9d45-84f7d2c192e0" containerName="cleanup" Jul 24 17:47:21.323478 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.323457 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.327289 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.327263 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-be21bfbc-openshift-default-dockercfg-tp7gt\"" Jul 24 17:47:21.336027 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.336005 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-podinfo\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.336152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.336032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-token\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.336152 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.336061 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b93a52-f5d2-4131-818e-89521eaa84f5-istiod-ca-cert\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.336257 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.336168 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-certs\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.336257 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.336197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-socket\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.336257 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.336224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-credential-socket\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.336257 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.336249 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-envoy\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.336461 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.336268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-data\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.336461 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.336296 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7fk2\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-kube-api-access-k7fk2\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.348083 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.348019 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j"] Jul 24 17:47:21.437143 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b93a52-f5d2-4131-818e-89521eaa84f5-istiod-ca-cert\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437337 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-certs\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437337 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-socket\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437337 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437242 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-credential-socket\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437337 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-envoy\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437337 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-data\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437337 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7fk2\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-kube-api-access-k7fk2\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437638 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-podinfo\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437638 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-token\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437638 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-certs\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.437817 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437633 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-credential-socket\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.438051 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.437946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b93a52-f5d2-4131-818e-89521eaa84f5-istiod-ca-cert\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.438163 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.438075 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-data\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.438225 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.438161 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-socket\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.440071 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.440047 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-envoy\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.440592 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.440570 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-podinfo\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.446260 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.446236 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-token\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.446393 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.446295 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7fk2\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-kube-api-access-k7fk2\") pod \"e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.635034 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.634948 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:21.783833 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.783790 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j"] Jul 24 17:47:21.785940 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:47:21.785899 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b93a52_f5d2_4131_818e_89521eaa84f5.slice/crio-353ca775274478d4351fd9e3e21d33b81919d7fce7c4aea0691c4ec67349daa8 WatchSource:0}: Error finding container 353ca775274478d4351fd9e3e21d33b81919d7fce7c4aea0691c4ec67349daa8: Status 404 returned error can't find the container with id 353ca775274478d4351fd9e3e21d33b81919d7fce7c4aea0691c4ec67349daa8 Jul 24 17:47:21.788392 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.788347 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:47:21.788468 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.788450 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:47:21.788514 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.788500 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:47:21.984307 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.984275 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" event={"ID":"45b93a52-f5d2-4131-818e-89521eaa84f5","Type":"ContainerStarted","Data":"3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9"} Jul 24 17:47:21.984515 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:21.984340 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" event={"ID":"45b93a52-f5d2-4131-818e-89521eaa84f5","Type":"ContainerStarted","Data":"353ca775274478d4351fd9e3e21d33b81919d7fce7c4aea0691c4ec67349daa8"} Jul 24 17:47:22.027194 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:22.027147 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" podStartSLOduration=1.027131363 podStartE2EDuration="1.027131363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:22.024532389 +0000 UTC m=+1798.690107026" watchObservedRunningTime="2026-07-24 17:47:22.027131363 +0000 UTC m=+1798.692706074" Jul 24 17:47:22.635202 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:22.635170 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:22.639890 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:22.639868 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:22.990715 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:22.990682 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:22.991551 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:22.991534 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:23.883696 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:23.883670 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:47:23.887400 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:23.887375 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:47:31.174278 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:31.174235 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:47:38.954292 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:38.954252 2574 patch_prober.go:28] interesting pod/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:51732->10.133.0.56:15021: read: connection reset by peer" start-of-body= Jul 24 17:47:38.954850 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:38.954338 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:51732->10.133.0.56:15021: read: connection reset by peer" Jul 24 17:47:39.088663 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:39.088632 2574 generic.go:356] "Generic (PLEG): container finished" podID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerID="3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9" exitCode=0 Jul 24 17:47:39.088847 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:39.088708 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" event={"ID":"45b93a52-f5d2-4131-818e-89521eaa84f5","Type":"ContainerDied","Data":"3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9"} Jul 24 17:47:39.089120 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:39.089105 2574 scope.go:117] "RemoveContainer" containerID="3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9" Jul 24 17:47:39.090226 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:39.090187 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:47:39.090347 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:39.090281 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:47:39.090347 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:39.090317 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:47:40.096947 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:40.096917 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" event={"ID":"45b93a52-f5d2-4131-818e-89521eaa84f5","Type":"ContainerStarted","Data":"5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea"} Jul 24 17:47:40.635586 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:40.635549 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:40.636265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:40.635941 2574 patch_prober.go:28] interesting pod/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 24 17:47:40.636265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:40.636008 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 24 17:47:41.636072 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:41.636034 2574 patch_prober.go:28] interesting pod/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 24 17:47:41.636553 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:41.636101 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 24 17:47:42.636306 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:42.636271 2574 patch_prober.go:28] interesting pod/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 24 17:47:42.636908 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:42.636876 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 24 17:47:43.636404 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:43.636356 2574 patch_prober.go:28] interesting pod/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 24 17:47:43.636929 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:43.636421 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 24 17:47:44.639375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:44.639344 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:44.639851 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:44.639621 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:44.640460 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:44.640440 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:47:49.062264 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:49.062226 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz"] Jul 24 17:47:49.064816 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:49.064800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" Jul 24 17:47:49.067760 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:49.067741 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:47:49.072752 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:49.072710 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz"] Jul 24 17:47:49.213462 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:49.213432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz\" (UID: \"c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" Jul 24 17:47:49.315028 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:49.314932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz\" (UID: \"c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" Jul 24 17:47:49.315530 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:49.315512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz\" (UID: \"c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" Jul 24 17:47:49.378124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:49.378082 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" Jul 24 17:47:49.513706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:49.513680 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz"] Jul 24 17:47:49.514805 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:47:49.514778 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c5f24b_8f5a_41f1_9dad_e1cd3c94e76a.slice/crio-4859d9437ff415ed623c139b50b8832bc25d1df2b68d8477fd2e965425d1ccf5 WatchSource:0}: Error finding container 4859d9437ff415ed623c139b50b8832bc25d1df2b68d8477fd2e965425d1ccf5: Status 404 returned error can't find the container with id 4859d9437ff415ed623c139b50b8832bc25d1df2b68d8477fd2e965425d1ccf5 Jul 24 17:47:50.159210 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:50.159063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" event={"ID":"c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a","Type":"ContainerStarted","Data":"a7b76a8e7af6b089c4f43987b80ad611cdce0d97f8a39e73bbca9a616a56a44e"} Jul 24 17:47:50.159210 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:50.159179 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" event={"ID":"c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a","Type":"ContainerStarted","Data":"4859d9437ff415ed623c139b50b8832bc25d1df2b68d8477fd2e965425d1ccf5"} Jul 24 17:47:50.178236 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:50.178185 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" podStartSLOduration=1.178170545 podStartE2EDuration="1.178170545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:50.175855215 +0000 UTC m=+1826.841429879" watchObservedRunningTime="2026-07-24 17:47:50.178170545 +0000 UTC m=+1826.843745182" Jul 24 17:47:51.166235 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:51.166203 2574 generic.go:356] "Generic (PLEG): container finished" podID="c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a" containerID="a7b76a8e7af6b089c4f43987b80ad611cdce0d97f8a39e73bbca9a616a56a44e" exitCode=0 Jul 24 17:47:51.166756 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:51.166297 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" event={"ID":"c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a","Type":"ContainerDied","Data":"a7b76a8e7af6b089c4f43987b80ad611cdce0d97f8a39e73bbca9a616a56a44e"} Jul 24 17:47:52.309404 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:52.309379 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" Jul 24 17:47:52.446160 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:52.446071 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a-maas-api-service-ca\") pod \"c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a\" (UID: \"c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a\") " Jul 24 17:47:52.446418 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:52.446397 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a" (UID: "c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:47:52.547451 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:52.547420 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:47:53.178829 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:53.178800 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" Jul 24 17:47:53.178829 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:53.178812 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz" event={"ID":"c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a","Type":"ContainerDied","Data":"4859d9437ff415ed623c139b50b8832bc25d1df2b68d8477fd2e965425d1ccf5"} Jul 24 17:47:53.179044 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:47:53.178856 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4859d9437ff415ed623c139b50b8832bc25d1df2b68d8477fd2e965425d1ccf5" Jul 24 17:48:02.360409 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:02.360367 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:48:04.180891 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:04.180860 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz"] Jul 24 17:48:04.184019 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:04.183992 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be21bfbc-4pjrz"] Jul 24 17:48:04.338841 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:04.338810 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j"] Jul 24 17:48:04.339115 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:04.339084 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" containerID="cri-o://5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea" gracePeriod=30 Jul 24 17:48:05.436954 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.436912 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z"] Jul 24 17:48:05.437406 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.437389 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a" containerName="revoke-keys" Jul 24 17:48:05.437455 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.437409 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a" containerName="revoke-keys" Jul 24 17:48:05.437507 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.437496 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a" containerName="revoke-keys" Jul 24 17:48:05.440845 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.440821 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.444786 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.444766 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ad794cce-gw-openshift-default-dockercfg-7t2lx\"" Jul 24 17:48:05.456026 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.456001 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z"] Jul 24 17:48:05.458749 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.458700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-podinfo\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.458862 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.458768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-token\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.458862 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.458796 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85c55\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-kube-api-access-85c55\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.458862 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.458850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-data\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.459046 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.458882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-envoy\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.459046 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.458903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istiod-ca-cert\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.459046 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.458989 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-credential-socket\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.459046 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.459035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-certs\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.459234 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.459153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-socket\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.560397 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.560361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-credential-socket\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.560397 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.560398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-certs\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.560661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.560451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-socket\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.560661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.560478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-podinfo\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.560661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.560504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-token\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.560661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.560519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85c55\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-kube-api-access-85c55\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.560661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.560551 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-data\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.560661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.560580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-envoy\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.560661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.560613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istiod-ca-cert\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.561235 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.561205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-certs\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.561355 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.561267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-socket\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.561435 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.561412 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-data\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.561493 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.561461 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-credential-socket\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.561638 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.561618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istiod-ca-cert\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.563407 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.563374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-envoy\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.563598 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.563582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-podinfo\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.570415 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.570384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-token\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.570510 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.570489 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85c55\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-kube-api-access-85c55\") pod \"e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.754770 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.754704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:05.801988 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.801956 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a" path="/var/lib/kubelet/pods/c8c5f24b-8f5a-41f1-9dad-e1cd3c94e76a/volumes" Jul 24 17:48:05.916665 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.909621 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z"] Jul 24 17:48:05.916665 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:48:05.916603 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13f0caa_ad03_4891_9b0e_644aa32abbd3.slice/crio-1c6724c1bfb55012c7cc656c52e4883d6da34f930e3e0bf35186215dc5632b7d WatchSource:0}: Error finding container 1c6724c1bfb55012c7cc656c52e4883d6da34f930e3e0bf35186215dc5632b7d: Status 404 returned error can't find the container with id 1c6724c1bfb55012c7cc656c52e4883d6da34f930e3e0bf35186215dc5632b7d Jul 24 17:48:05.918866 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.918828 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:48:05.918965 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.918909 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:48:05.918965 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:05.918952 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:48:06.261932 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:06.261889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" event={"ID":"b13f0caa-ad03-4891-9b0e-644aa32abbd3","Type":"ContainerStarted","Data":"c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261"} Jul 24 17:48:06.262145 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:06.261950 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" event={"ID":"b13f0caa-ad03-4891-9b0e-644aa32abbd3","Type":"ContainerStarted","Data":"1c6724c1bfb55012c7cc656c52e4883d6da34f930e3e0bf35186215dc5632b7d"} Jul 24 17:48:06.285814 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:06.285763 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" podStartSLOduration=1.28574571 podStartE2EDuration="1.28574571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:48:06.283206079 +0000 UTC m=+1842.948780717" watchObservedRunningTime="2026-07-24 17:48:06.28574571 +0000 UTC m=+1842.951320339" Jul 24 17:48:06.755429 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:06.755392 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:06.760521 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:06.760498 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:07.268913 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:07.268886 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:07.269871 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:07.269851 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:09.370636 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:48:09.370609 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b93a52_f5d2_4131_818e_89521eaa84f5.slice/crio-5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea.scope\": RecentStats: unable to find data in memory cache]" Jul 24 17:48:09.602693 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.602636 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:48:09.700049 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700015 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-podinfo\") pod \"45b93a52-f5d2-4131-818e-89521eaa84f5\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " Jul 24 17:48:09.700244 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700058 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-socket\") pod \"45b93a52-f5d2-4131-818e-89521eaa84f5\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " Jul 24 17:48:09.700244 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700086 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-envoy\") pod \"45b93a52-f5d2-4131-818e-89521eaa84f5\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " Jul 24 17:48:09.700244 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700149 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-certs\") pod \"45b93a52-f5d2-4131-818e-89521eaa84f5\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " Jul 24 17:48:09.700244 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700183 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b93a52-f5d2-4131-818e-89521eaa84f5-istiod-ca-cert\") pod \"45b93a52-f5d2-4131-818e-89521eaa84f5\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " Jul 24 17:48:09.700244 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700210 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-credential-socket\") pod \"45b93a52-f5d2-4131-818e-89521eaa84f5\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " Jul 24 17:48:09.700244 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700239 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-token\") pod \"45b93a52-f5d2-4131-818e-89521eaa84f5\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " Jul 24 17:48:09.700560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700287 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7fk2\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-kube-api-access-k7fk2\") pod \"45b93a52-f5d2-4131-818e-89521eaa84f5\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " Jul 24 17:48:09.700560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700365 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45b93a52-f5d2-4131-818e-89521eaa84f5" (UID: "45b93a52-f5d2-4131-818e-89521eaa84f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:09.700560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700389 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-data\") pod \"45b93a52-f5d2-4131-818e-89521eaa84f5\" (UID: \"45b93a52-f5d2-4131-818e-89521eaa84f5\") " Jul 24 17:48:09.700560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700494 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45b93a52-f5d2-4131-818e-89521eaa84f5" (UID: "45b93a52-f5d2-4131-818e-89521eaa84f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:09.700811 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700705 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:09.700811 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700757 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:09.700811 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.700672 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b93a52-f5d2-4131-818e-89521eaa84f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45b93a52-f5d2-4131-818e-89521eaa84f5" (UID: "45b93a52-f5d2-4131-818e-89521eaa84f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:48:09.701133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.701110 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45b93a52-f5d2-4131-818e-89521eaa84f5" (UID: "45b93a52-f5d2-4131-818e-89521eaa84f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:09.701393 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.701373 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "45b93a52-f5d2-4131-818e-89521eaa84f5" (UID: "45b93a52-f5d2-4131-818e-89521eaa84f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:09.702441 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.702418 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45b93a52-f5d2-4131-818e-89521eaa84f5" (UID: "45b93a52-f5d2-4131-818e-89521eaa84f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:48:09.702571 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.702554 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "45b93a52-f5d2-4131-818e-89521eaa84f5" (UID: "45b93a52-f5d2-4131-818e-89521eaa84f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:48:09.702972 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.702954 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45b93a52-f5d2-4131-818e-89521eaa84f5" (UID: "45b93a52-f5d2-4131-818e-89521eaa84f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:09.703044 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.703010 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-kube-api-access-k7fk2" (OuterVolumeSpecName: "kube-api-access-k7fk2") pod "45b93a52-f5d2-4131-818e-89521eaa84f5" (UID: "45b93a52-f5d2-4131-818e-89521eaa84f5"). InnerVolumeSpecName "kube-api-access-k7fk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:48:09.801169 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.801141 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:09.801169 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.801165 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:09.801169 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.801175 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:09.801392 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.801182 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b93a52-f5d2-4131-818e-89521eaa84f5-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:09.801392 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.801190 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b93a52-f5d2-4131-818e-89521eaa84f5-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:09.801392 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.801200 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:09.801392 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:09.801208 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7fk2\" (UniqueName: \"kubernetes.io/projected/45b93a52-f5d2-4131-818e-89521eaa84f5-kube-api-access-k7fk2\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:10.287780 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.287742 2574 generic.go:356] "Generic (PLEG): container finished" podID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerID="5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea" exitCode=0 Jul 24 17:48:10.287780 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.287781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" event={"ID":"45b93a52-f5d2-4131-818e-89521eaa84f5","Type":"ContainerDied","Data":"5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea"} Jul 24 17:48:10.288053 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.287824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" event={"ID":"45b93a52-f5d2-4131-818e-89521eaa84f5","Type":"ContainerDied","Data":"353ca775274478d4351fd9e3e21d33b81919d7fce7c4aea0691c4ec67349daa8"} Jul 24 17:48:10.288053 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.287824 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j" Jul 24 17:48:10.288053 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.287845 2574 scope.go:117] "RemoveContainer" containerID="5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea" Jul 24 17:48:10.299326 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.299300 2574 scope.go:117] "RemoveContainer" containerID="3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9" Jul 24 17:48:10.308526 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.308502 2574 scope.go:117] "RemoveContainer" containerID="5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea" Jul 24 17:48:10.308827 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:48:10.308804 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea\": container with ID starting with 5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea not found: ID does not exist" containerID="5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea" Jul 24 17:48:10.308876 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.308836 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea"} err="failed to get container status \"5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea\": rpc error: code = NotFound desc = could not find container \"5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea\": container with ID starting with 5ef950beb0543e821a79d0c222706ae5fc266670bf65650654d8aaebbfaa38ea not found: ID does not exist" Jul 24 17:48:10.308876 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.308852 2574 scope.go:117] "RemoveContainer" containerID="3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9" Jul 24 17:48:10.309117 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:48:10.309093 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9\": container with ID starting with 3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9 not found: ID does not exist" containerID="3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9" Jul 24 17:48:10.309172 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.309125 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9"} err="failed to get container status \"3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9\": rpc error: code = NotFound desc = could not find container \"3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9\": container with ID starting with 3f8fe5122d858b2f4090011413fd984f659ce1ce3845d3dffbec698396bd8ab9 not found: ID does not exist" Jul 24 17:48:10.309994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.309972 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j"] Jul 24 17:48:10.313504 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:10.313483 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-be21bfbc-openshift-default-67bc684c7d-qth5j"] Jul 24 17:48:11.801557 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:11.801516 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" path="/var/lib/kubelet/pods/45b93a52-f5d2-4131-818e-89521eaa84f5/volumes" Jul 24 17:48:13.778523 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:13.778487 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:48:22.270163 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:22.270123 2574 patch_prober.go:28] interesting pod/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 24 17:48:22.270576 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:22.270191 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 24 17:48:22.359926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:22.359887 2574 generic.go:356] "Generic (PLEG): container finished" podID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerID="c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261" exitCode=0 Jul 24 17:48:22.360088 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:22.359961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" event={"ID":"b13f0caa-ad03-4891-9b0e-644aa32abbd3","Type":"ContainerDied","Data":"c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261"} Jul 24 17:48:22.360349 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:22.360335 2574 scope.go:117] "RemoveContainer" containerID="c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261" Jul 24 17:48:22.361347 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:22.361317 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:48:22.361451 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:22.361385 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:48:22.361451 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:22.361423 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:48:23.367826 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:23.367788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" event={"ID":"b13f0caa-ad03-4891-9b0e-644aa32abbd3","Type":"ContainerStarted","Data":"339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0"} Jul 24 17:48:23.755680 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:23.755183 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:23.755680 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:23.755316 2574 patch_prober.go:28] interesting pod/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 24 17:48:23.755680 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:23.755380 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 24 17:48:24.755482 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:24.755443 2574 patch_prober.go:28] interesting pod/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 24 17:48:24.756118 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:24.756083 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 24 17:48:25.755217 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:25.755179 2574 patch_prober.go:28] interesting pod/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 24 17:48:25.755547 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:25.755508 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 24 17:48:26.755918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:26.755872 2574 patch_prober.go:28] interesting pod/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 24 17:48:26.756457 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:26.755944 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 24 17:48:27.760148 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:27.760115 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:27.760560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:27.760398 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:27.761053 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:27.761035 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:32.878623 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.878582 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8"] Jul 24 17:48:32.879029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.878981 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" Jul 24 17:48:32.879029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.878993 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" Jul 24 17:48:32.879029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.879001 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" Jul 24 17:48:32.879029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.879007 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" Jul 24 17:48:32.879163 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.879079 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" Jul 24 17:48:32.879163 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.879094 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b93a52-f5d2-4131-818e-89521eaa84f5" containerName="istio-proxy" Jul 24 17:48:32.882225 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.882209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" Jul 24 17:48:32.886522 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.886501 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:48:32.900163 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:32.900137 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8"] Jul 24 17:48:33.033018 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:33.032977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1a0315-2f3d-4139-acc1-566d3fe911e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8\" (UID: \"dd1a0315-2f3d-4139-acc1-566d3fe911e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" Jul 24 17:48:33.133874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:33.133790 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1a0315-2f3d-4139-acc1-566d3fe911e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8\" (UID: \"dd1a0315-2f3d-4139-acc1-566d3fe911e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" Jul 24 17:48:33.134398 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:33.134381 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1a0315-2f3d-4139-acc1-566d3fe911e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8\" (UID: \"dd1a0315-2f3d-4139-acc1-566d3fe911e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" Jul 24 17:48:33.192561 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:33.192533 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" Jul 24 17:48:33.333301 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:33.333272 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8"] Jul 24 17:48:33.334937 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:48:33.334909 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1a0315_2f3d_4139_acc1_566d3fe911e5.slice/crio-dd1dce53459cfbffff8e337d564350d806e413dc5a9758ed76452312466b1070 WatchSource:0}: Error finding container dd1dce53459cfbffff8e337d564350d806e413dc5a9758ed76452312466b1070: Status 404 returned error can't find the container with id dd1dce53459cfbffff8e337d564350d806e413dc5a9758ed76452312466b1070 Jul 24 17:48:33.438140 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:33.438053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" event={"ID":"dd1a0315-2f3d-4139-acc1-566d3fe911e5","Type":"ContainerStarted","Data":"dd1dce53459cfbffff8e337d564350d806e413dc5a9758ed76452312466b1070"} Jul 24 17:48:34.445327 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:34.445292 2574 generic.go:356] "Generic (PLEG): container finished" podID="dd1a0315-2f3d-4139-acc1-566d3fe911e5" containerID="2efa99a376e427c4bfb8cfeb76520dedc8105b40cd506ba89395c94ce9e179ca" exitCode=0 Jul 24 17:48:34.445703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:34.445383 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" event={"ID":"dd1a0315-2f3d-4139-acc1-566d3fe911e5","Type":"ContainerDied","Data":"2efa99a376e427c4bfb8cfeb76520dedc8105b40cd506ba89395c94ce9e179ca"} Jul 24 17:48:35.593886 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:35.593858 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" Jul 24 17:48:35.759344 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:35.759310 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1a0315-2f3d-4139-acc1-566d3fe911e5-maas-api-service-ca\") pod \"dd1a0315-2f3d-4139-acc1-566d3fe911e5\" (UID: \"dd1a0315-2f3d-4139-acc1-566d3fe911e5\") " Jul 24 17:48:35.759683 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:35.759657 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1a0315-2f3d-4139-acc1-566d3fe911e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd1a0315-2f3d-4139-acc1-566d3fe911e5" (UID: "dd1a0315-2f3d-4139-acc1-566d3fe911e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:48:35.860239 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:35.860206 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1a0315-2f3d-4139-acc1-566d3fe911e5-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:36.458573 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:36.458547 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" Jul 24 17:48:36.458573 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:36.458555 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8" event={"ID":"dd1a0315-2f3d-4139-acc1-566d3fe911e5","Type":"ContainerDied","Data":"dd1dce53459cfbffff8e337d564350d806e413dc5a9758ed76452312466b1070"} Jul 24 17:48:36.458800 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:36.458604 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd1dce53459cfbffff8e337d564350d806e413dc5a9758ed76452312466b1070" Jul 24 17:48:46.468328 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:46.468291 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:48:48.004711 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:48.004672 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8"] Jul 24 17:48:48.009279 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:48.009257 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad794cce-9whs8"] Jul 24 17:48:48.181361 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:48.181329 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z"] Jul 24 17:48:48.181594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:48.181572 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" containerID="cri-o://339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0" gracePeriod=30 Jul 24 17:48:49.113178 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.113136 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87"] Jul 24 17:48:49.113560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.113531 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1a0315-2f3d-4139-acc1-566d3fe911e5" containerName="revoke-keys" Jul 24 17:48:49.113560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.113542 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1a0315-2f3d-4139-acc1-566d3fe911e5" containerName="revoke-keys" Jul 24 17:48:49.113633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.113621 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd1a0315-2f3d-4139-acc1-566d3fe911e5" containerName="revoke-keys" Jul 24 17:48:49.116973 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.116957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.120891 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.120868 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-647df330-openshift-default-dockercfg-hlcn8\"" Jul 24 17:48:49.135904 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.135830 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87"] Jul 24 17:48:49.173369 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.173334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-podinfo\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.173369 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.173372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-token\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.173594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.173395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istiod-ca-cert\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.173594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.173417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jp6v\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-kube-api-access-9jp6v\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.173594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.173443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-certs\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.173594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.173474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-socket\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.173594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.173488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-credential-socket\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.173594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.173560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-data\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.173594 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.173592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-envoy\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274219 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-podinfo\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274219 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-token\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274489 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istiod-ca-cert\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274489 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jp6v\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-kube-api-access-9jp6v\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274489 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-certs\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274489 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-socket\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274489 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-credential-socket\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-data\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-envoy\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274912 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274889 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-socket\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.274982 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.274957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-certs\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.275040 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.275005 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-data\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.275142 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.275121 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istiod-ca-cert\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.275188 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.275153 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-credential-socket\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.277104 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.277084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-podinfo\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.277227 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.277208 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-envoy\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.283850 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.283829 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-token\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.284011 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.283994 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jp6v\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-kube-api-access-9jp6v\") pod \"e2e-ait-647df330-openshift-default-6c894465d-4wx87\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.428189 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.428100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:49.585635 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.585605 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87"] Jul 24 17:48:49.586852 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:48:49.586819 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d13fe6e_7ec1_45b8_aa24_86408f8a2cec.slice/crio-7237a64dad207a979b6a2334b447011a2b21f8b365c43ea8975d5f89d4db3588 WatchSource:0}: Error finding container 7237a64dad207a979b6a2334b447011a2b21f8b365c43ea8975d5f89d4db3588: Status 404 returned error can't find the container with id 7237a64dad207a979b6a2334b447011a2b21f8b365c43ea8975d5f89d4db3588 Jul 24 17:48:49.589157 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.589123 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:48:49.589246 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.589199 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:48:49.589246 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.589228 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:48:49.803394 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:49.803357 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd1a0315-2f3d-4139-acc1-566d3fe911e5" path="/var/lib/kubelet/pods/dd1a0315-2f3d-4139-acc1-566d3fe911e5/volumes" Jul 24 17:48:50.542164 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:50.542126 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" event={"ID":"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec","Type":"ContainerStarted","Data":"42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4"} Jul 24 17:48:50.542625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:50.542186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" event={"ID":"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec","Type":"ContainerStarted","Data":"7237a64dad207a979b6a2334b447011a2b21f8b365c43ea8975d5f89d4db3588"} Jul 24 17:48:50.568485 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:50.568431 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" podStartSLOduration=1.5684136340000001 podStartE2EDuration="1.568413634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:48:50.566826649 +0000 UTC m=+1887.232401286" watchObservedRunningTime="2026-07-24 17:48:50.568413634 +0000 UTC m=+1887.233988271" Jul 24 17:48:51.428794 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:51.428759 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:51.433814 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:51.433787 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:51.553202 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:51.553170 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:51.554269 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:51.554252 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:48:53.445116 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.445057 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:53.515126 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515094 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istiod-ca-cert\") pod \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " Jul 24 17:48:53.515283 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515152 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-socket\") pod \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " Jul 24 17:48:53.515283 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515183 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-token\") pod \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " Jul 24 17:48:53.515283 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515220 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-envoy\") pod \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " Jul 24 17:48:53.515410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515316 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85c55\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-kube-api-access-85c55\") pod \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " Jul 24 17:48:53.515410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515354 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-credential-socket\") pod \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " Jul 24 17:48:53.515410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515390 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-data\") pod \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " Jul 24 17:48:53.515559 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515509 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b13f0caa-ad03-4891-9b0e-644aa32abbd3" (UID: "b13f0caa-ad03-4891-9b0e-644aa32abbd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:53.515622 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515582 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b13f0caa-ad03-4891-9b0e-644aa32abbd3" (UID: "b13f0caa-ad03-4891-9b0e-644aa32abbd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:48:53.515622 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515608 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b13f0caa-ad03-4891-9b0e-644aa32abbd3" (UID: "b13f0caa-ad03-4891-9b0e-644aa32abbd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:53.515759 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515619 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-certs\") pod \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " Jul 24 17:48:53.515759 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515677 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-podinfo\") pod \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\" (UID: \"b13f0caa-ad03-4891-9b0e-644aa32abbd3\") " Jul 24 17:48:53.515953 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.515895 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b13f0caa-ad03-4891-9b0e-644aa32abbd3" (UID: "b13f0caa-ad03-4891-9b0e-644aa32abbd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:53.516236 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.516206 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-data" (OuterVolumeSpecName: "istio-data") pod "b13f0caa-ad03-4891-9b0e-644aa32abbd3" (UID: "b13f0caa-ad03-4891-9b0e-644aa32abbd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:53.516236 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.516217 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:53.516362 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.516255 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:53.516362 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.516274 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:53.516362 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.516287 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:53.517609 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.517581 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-token" (OuterVolumeSpecName: "istio-token") pod "b13f0caa-ad03-4891-9b0e-644aa32abbd3" (UID: "b13f0caa-ad03-4891-9b0e-644aa32abbd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:48:53.517782 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.517686 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b13f0caa-ad03-4891-9b0e-644aa32abbd3" (UID: "b13f0caa-ad03-4891-9b0e-644aa32abbd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:53.517782 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.517745 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-kube-api-access-85c55" (OuterVolumeSpecName: "kube-api-access-85c55") pod "b13f0caa-ad03-4891-9b0e-644aa32abbd3" (UID: "b13f0caa-ad03-4891-9b0e-644aa32abbd3"). InnerVolumeSpecName "kube-api-access-85c55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:48:53.518024 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.518005 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b13f0caa-ad03-4891-9b0e-644aa32abbd3" (UID: "b13f0caa-ad03-4891-9b0e-644aa32abbd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:48:53.566756 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.566700 2574 generic.go:356] "Generic (PLEG): container finished" podID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerID="339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0" exitCode=0 Jul 24 17:48:53.566853 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.566774 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" Jul 24 17:48:53.566853 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.566763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" event={"ID":"b13f0caa-ad03-4891-9b0e-644aa32abbd3","Type":"ContainerDied","Data":"339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0"} Jul 24 17:48:53.566853 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.566836 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z" event={"ID":"b13f0caa-ad03-4891-9b0e-644aa32abbd3","Type":"ContainerDied","Data":"1c6724c1bfb55012c7cc656c52e4883d6da34f930e3e0bf35186215dc5632b7d"} Jul 24 17:48:53.566987 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.566865 2574 scope.go:117] "RemoveContainer" containerID="339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0" Jul 24 17:48:53.576779 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.576719 2574 scope.go:117] "RemoveContainer" containerID="c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261" Jul 24 17:48:53.585520 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.585504 2574 scope.go:117] "RemoveContainer" containerID="339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0" Jul 24 17:48:53.585773 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:48:53.585756 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0\": container with ID starting with 339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0 not found: ID does not exist" containerID="339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0" Jul 24 17:48:53.585848 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.585781 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0"} err="failed to get container status \"339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0\": rpc error: code = NotFound desc = could not find container \"339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0\": container with ID starting with 339907338da7c8682e25fb538d2e110ec3425722d555895f18b4f98fe44df8d0 not found: ID does not exist" Jul 24 17:48:53.585848 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.585796 2574 scope.go:117] "RemoveContainer" containerID="c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261" Jul 24 17:48:53.586041 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:48:53.586026 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261\": container with ID starting with c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261 not found: ID does not exist" containerID="c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261" Jul 24 17:48:53.586081 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.586044 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261"} err="failed to get container status \"c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261\": rpc error: code = NotFound desc = could not find container \"c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261\": container with ID starting with c5248b14d8eb40c79fcfdf19be37db66b46dae91912ede9e4ae5a78c121b3261 not found: ID does not exist" Jul 24 17:48:53.595267 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.595245 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z"] Jul 24 17:48:53.605387 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.605365 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ad794cce-gw-openshift-default-99868c6c6-8bh6z"] Jul 24 17:48:53.616818 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.616791 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:53.616818 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.616814 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:53.616956 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.616826 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:53.616956 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.616837 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13f0caa-ad03-4891-9b0e-644aa32abbd3-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:53.616956 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.616850 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85c55\" (UniqueName: \"kubernetes.io/projected/b13f0caa-ad03-4891-9b0e-644aa32abbd3-kube-api-access-85c55\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:48:53.801861 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:53.801828 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" path="/var/lib/kubelet/pods/b13f0caa-ad03-4891-9b0e-644aa32abbd3/volumes" Jul 24 17:48:58.182832 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:48:58.182801 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:49:06.554622 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:06.554585 2574 patch_prober.go:28] interesting pod/e2e-ait-647df330-openshift-default-6c894465d-4wx87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 17:49:06.555077 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:06.554646 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 17:49:06.647738 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:06.647703 2574 generic.go:356] "Generic (PLEG): container finished" podID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerID="42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4" exitCode=0 Jul 24 17:49:06.647916 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:06.647777 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" event={"ID":"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec","Type":"ContainerDied","Data":"42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4"} Jul 24 17:49:06.648180 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:06.648167 2574 scope.go:117] "RemoveContainer" containerID="42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4" Jul 24 17:49:06.649233 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:06.649207 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:49:06.649290 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:06.649274 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:49:06.649326 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:06.649307 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:49:07.660376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:07.660343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" event={"ID":"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec","Type":"ContainerStarted","Data":"5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3"} Jul 24 17:49:08.428886 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:08.428856 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:49:08.429085 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:08.429040 2574 patch_prober.go:28] interesting pod/e2e-ait-647df330-openshift-default-6c894465d-4wx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 17:49:08.429159 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:08.429097 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 17:49:09.429439 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:09.429397 2574 patch_prober.go:28] interesting pod/e2e-ait-647df330-openshift-default-6c894465d-4wx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 17:49:09.429875 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:09.429467 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 17:49:10.429300 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:10.429264 2574 patch_prober.go:28] interesting pod/e2e-ait-647df330-openshift-default-6c894465d-4wx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 17:49:10.429480 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:10.429318 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 17:49:11.428566 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:11.428530 2574 patch_prober.go:28] interesting pod/e2e-ait-647df330-openshift-default-6c894465d-4wx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 24 17:49:11.428769 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:11.428586 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 24 17:49:12.432706 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:12.432676 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:49:12.433088 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:12.432944 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:49:12.433684 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:12.433669 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:49:17.875815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:17.875779 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp"] Jul 24 17:49:17.876227 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:17.876153 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" Jul 24 17:49:17.876227 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:17.876163 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" Jul 24 17:49:17.876306 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:17.876242 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" Jul 24 17:49:17.879453 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:17.879431 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" Jul 24 17:49:17.882514 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:17.882494 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:49:17.889526 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:17.889502 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp"] Jul 24 17:49:18.033926 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:18.033873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829db227-da5c-4d20-b856-ebd4485c03bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-647df330-9z6jp\" (UID: \"829db227-da5c-4d20-b856-ebd4485c03bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" Jul 24 17:49:18.134481 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:18.134398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829db227-da5c-4d20-b856-ebd4485c03bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-647df330-9z6jp\" (UID: \"829db227-da5c-4d20-b856-ebd4485c03bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" Jul 24 17:49:18.135074 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:18.135052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829db227-da5c-4d20-b856-ebd4485c03bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-647df330-9z6jp\" (UID: \"829db227-da5c-4d20-b856-ebd4485c03bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" Jul 24 17:49:18.191340 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:18.191311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" Jul 24 17:49:18.333809 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:18.333781 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp"] Jul 24 17:49:18.335598 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:49:18.335570 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829db227_da5c_4d20_b856_ebd4485c03bc.slice/crio-60f05129c1f4e075b5c2d58080dd725516f02d7b3596e8bcc5adcca4badb62c2 WatchSource:0}: Error finding container 60f05129c1f4e075b5c2d58080dd725516f02d7b3596e8bcc5adcca4badb62c2: Status 404 returned error can't find the container with id 60f05129c1f4e075b5c2d58080dd725516f02d7b3596e8bcc5adcca4badb62c2 Jul 24 17:49:18.722802 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:18.722762 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" event={"ID":"829db227-da5c-4d20-b856-ebd4485c03bc","Type":"ContainerStarted","Data":"354ecead28f54ac1486fd9c6673fe9ffec90ab944aa47d4556969122d646bae7"} Jul 24 17:49:18.722998 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:18.722831 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" event={"ID":"829db227-da5c-4d20-b856-ebd4485c03bc","Type":"ContainerStarted","Data":"60f05129c1f4e075b5c2d58080dd725516f02d7b3596e8bcc5adcca4badb62c2"} Jul 24 17:49:18.742763 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:18.742697 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" podStartSLOduration=1.742680706 podStartE2EDuration="1.742680706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:18.739041707 +0000 UTC m=+1915.404616345" watchObservedRunningTime="2026-07-24 17:49:18.742680706 +0000 UTC m=+1915.408255345" Jul 24 17:49:19.729970 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:19.729937 2574 generic.go:356] "Generic (PLEG): container finished" podID="829db227-da5c-4d20-b856-ebd4485c03bc" containerID="354ecead28f54ac1486fd9c6673fe9ffec90ab944aa47d4556969122d646bae7" exitCode=0 Jul 24 17:49:19.730384 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:19.730030 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" event={"ID":"829db227-da5c-4d20-b856-ebd4485c03bc","Type":"ContainerDied","Data":"354ecead28f54ac1486fd9c6673fe9ffec90ab944aa47d4556969122d646bae7"} Jul 24 17:49:20.876201 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:20.876177 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" Jul 24 17:49:20.959302 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:20.959271 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829db227-da5c-4d20-b856-ebd4485c03bc-maas-api-service-ca\") pod \"829db227-da5c-4d20-b856-ebd4485c03bc\" (UID: \"829db227-da5c-4d20-b856-ebd4485c03bc\") " Jul 24 17:49:20.959661 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:20.959640 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829db227-da5c-4d20-b856-ebd4485c03bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "829db227-da5c-4d20-b856-ebd4485c03bc" (UID: "829db227-da5c-4d20-b856-ebd4485c03bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:49:21.060635 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:21.060563 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/829db227-da5c-4d20-b856-ebd4485c03bc-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:21.743758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:21.743712 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" Jul 24 17:49:21.743933 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:21.743757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp" event={"ID":"829db227-da5c-4d20-b856-ebd4485c03bc","Type":"ContainerDied","Data":"60f05129c1f4e075b5c2d58080dd725516f02d7b3596e8bcc5adcca4badb62c2"} Jul 24 17:49:21.743933 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:21.743814 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f05129c1f4e075b5c2d58080dd725516f02d7b3596e8bcc5adcca4badb62c2" Jul 24 17:49:31.571530 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:31.571449 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:49:32.958500 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:32.958458 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp"] Jul 24 17:49:32.964371 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:32.964347 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-647df330-9z6jp"] Jul 24 17:49:33.801652 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:33.801620 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="829db227-da5c-4d20-b856-ebd4485c03bc" path="/var/lib/kubelet/pods/829db227-da5c-4d20-b856-ebd4485c03bc/volumes" Jul 24 17:49:34.295628 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:34.295601 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87"] Jul 24 17:49:34.296074 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:34.295815 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" containerID="cri-o://5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3" gracePeriod=30 Jul 24 17:49:35.091707 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.091675 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv"] Jul 24 17:49:35.092102 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.092087 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="829db227-da5c-4d20-b856-ebd4485c03bc" containerName="revoke-keys" Jul 24 17:49:35.092155 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.092104 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="829db227-da5c-4d20-b856-ebd4485c03bc" containerName="revoke-keys" Jul 24 17:49:35.092155 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.092122 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" Jul 24 17:49:35.092155 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.092128 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" Jul 24 17:49:35.092290 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.092194 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="829db227-da5c-4d20-b856-ebd4485c03bc" containerName="revoke-keys" Jul 24 17:49:35.092290 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.092202 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b13f0caa-ad03-4891-9b0e-644aa32abbd3" containerName="istio-proxy" Jul 24 17:49:35.095549 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.095532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.099337 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.099316 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e7442c1c-openshift-default-dockercfg-mp475\"" Jul 24 17:49:35.110715 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.110685 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv"] Jul 24 17:49:35.184060 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.184016 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-socket\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.184265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.184082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639b5027-5404-4d24-9b40-16faad8046fb-istio-podinfo\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.184265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.184120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-data\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.184265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.184142 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-credential-socket\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.184265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.184219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-istio-token\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.184265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.184258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-certs\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.184527 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.184278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639b5027-5404-4d24-9b40-16faad8046fb-istiod-ca-cert\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.184527 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.184333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-envoy\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.184527 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.184378 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8jr\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-kube-api-access-rz8jr\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-data\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-credential-socket\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-istio-token\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-certs\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639b5027-5404-4d24-9b40-16faad8046fb-istiod-ca-cert\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-envoy\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8jr\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-kube-api-access-rz8jr\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-socket\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639b5027-5404-4d24-9b40-16faad8046fb-istio-podinfo\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.285994 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-data\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.286230 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.285998 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-certs\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.286230 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.286183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-socket\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.286344 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.286329 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-credential-socket\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.286450 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.286424 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639b5027-5404-4d24-9b40-16faad8046fb-istiod-ca-cert\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.288281 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.288262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-envoy\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.288658 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.288641 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639b5027-5404-4d24-9b40-16faad8046fb-istio-podinfo\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.294904 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.294874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-istio-token\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.295019 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.294919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8jr\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-kube-api-access-rz8jr\") pod \"e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.406213 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.405951 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:35.562232 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.562205 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv"] Jul 24 17:49:35.563143 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:49:35.563116 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639b5027_5404_4d24_9b40_16faad8046fb.slice/crio-806cc145aa4ac512c5ef43b3a98b2d5cf610d7b79aabd72aa40b68dfa28d2822 WatchSource:0}: Error finding container 806cc145aa4ac512c5ef43b3a98b2d5cf610d7b79aabd72aa40b68dfa28d2822: Status 404 returned error can't find the container with id 806cc145aa4ac512c5ef43b3a98b2d5cf610d7b79aabd72aa40b68dfa28d2822 Jul 24 17:49:35.565294 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.565262 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:49:35.565367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.565326 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:49:35.565367 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.565354 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:49:35.836774 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.836699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" event={"ID":"639b5027-5404-4d24-9b40-16faad8046fb","Type":"ContainerStarted","Data":"b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df"} Jul 24 17:49:35.836966 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.836802 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" event={"ID":"639b5027-5404-4d24-9b40-16faad8046fb","Type":"ContainerStarted","Data":"806cc145aa4ac512c5ef43b3a98b2d5cf610d7b79aabd72aa40b68dfa28d2822"} Jul 24 17:49:35.861618 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:35.861568 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" podStartSLOduration=0.861552118 podStartE2EDuration="861.552118ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:35.857914985 +0000 UTC m=+1932.523489646" watchObservedRunningTime="2026-07-24 17:49:35.861552118 +0000 UTC m=+1932.527126755" Jul 24 17:49:36.406332 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:36.406297 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:36.411742 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:36.411700 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:36.842636 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:36.842602 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:36.843659 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:36.843643 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:39.547262 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.547242 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:49:39.626183 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626098 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-credential-socket\") pod \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " Jul 24 17:49:39.626183 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626151 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jp6v\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-kube-api-access-9jp6v\") pod \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " Jul 24 17:49:39.626183 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626171 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-envoy\") pod \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " Jul 24 17:49:39.626464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626211 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istiod-ca-cert\") pod \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " Jul 24 17:49:39.626464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626231 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-certs\") pod \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " Jul 24 17:49:39.626464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626250 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-data\") pod \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " Jul 24 17:49:39.626464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626331 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-podinfo\") pod \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " Jul 24 17:49:39.626464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626358 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-token\") pod \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " Jul 24 17:49:39.626464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626377 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-socket\") pod \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\" (UID: \"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec\") " Jul 24 17:49:39.626464 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626379 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" (UID: "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:49:39.626858 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626651 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:39.626858 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626701 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" (UID: "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:49:39.627133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626890 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" (UID: "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:49:39.627133 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.626905 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" (UID: "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:49:39.627486 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.627460 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" (UID: "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:49:39.628621 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.628591 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" (UID: "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:49:39.628621 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.628598 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-kube-api-access-9jp6v" (OuterVolumeSpecName: "kube-api-access-9jp6v") pod "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" (UID: "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec"). InnerVolumeSpecName "kube-api-access-9jp6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:49:39.628801 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.628634 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" (UID: "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:49:39.628905 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.628888 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" (UID: "0d13fe6e-7ec1-45b8-aa24-86408f8a2cec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:49:39.727512 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.727481 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:39.727512 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.727503 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:39.727512 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.727512 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:39.727512 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.727519 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:39.727874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.727527 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:39.727874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.727534 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:39.727874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.727542 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jp6v\" (UniqueName: \"kubernetes.io/projected/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-kube-api-access-9jp6v\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:39.727874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.727550 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:49:39.861699 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.861670 2574 generic.go:356] "Generic (PLEG): container finished" podID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerID="5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3" exitCode=0 Jul 24 17:49:39.861860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.861716 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" Jul 24 17:49:39.861860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.861714 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" event={"ID":"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec","Type":"ContainerDied","Data":"5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3"} Jul 24 17:49:39.861860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.861796 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87" event={"ID":"0d13fe6e-7ec1-45b8-aa24-86408f8a2cec","Type":"ContainerDied","Data":"7237a64dad207a979b6a2334b447011a2b21f8b365c43ea8975d5f89d4db3588"} Jul 24 17:49:39.861860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.861828 2574 scope.go:117] "RemoveContainer" containerID="5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3" Jul 24 17:49:39.871659 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.871644 2574 scope.go:117] "RemoveContainer" containerID="42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4" Jul 24 17:49:39.880105 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.880084 2574 scope.go:117] "RemoveContainer" containerID="5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3" Jul 24 17:49:39.880370 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:49:39.880349 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3\": container with ID starting with 5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3 not found: ID does not exist" containerID="5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3" Jul 24 17:49:39.880440 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.880383 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3"} err="failed to get container status \"5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3\": rpc error: code = NotFound desc = could not find container \"5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3\": container with ID starting with 5eb1965508a0d99a393442e777bd0f762ef80440624f234204464c6984bc38a3 not found: ID does not exist" Jul 24 17:49:39.880440 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.880409 2574 scope.go:117] "RemoveContainer" containerID="42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4" Jul 24 17:49:39.880676 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:49:39.880659 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4\": container with ID starting with 42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4 not found: ID does not exist" containerID="42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4" Jul 24 17:49:39.880716 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.880682 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4"} err="failed to get container status \"42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4\": rpc error: code = NotFound desc = could not find container \"42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4\": container with ID starting with 42a3954312ee4ed04eb5ba1d3a68b9d96ddb027253ac0cbff9bf329dd9a1cbe4 not found: ID does not exist" Jul 24 17:49:39.882368 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.882350 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87"] Jul 24 17:49:39.887288 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:39.887270 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-647df330-openshift-default-6c894465d-4wx87"] Jul 24 17:49:41.801871 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:41.801839 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" path="/var/lib/kubelet/pods/0d13fe6e-7ec1-45b8-aa24-86408f8a2cec/volumes" Jul 24 17:49:42.872959 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:42.868431 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:49:50.927886 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:50.927849 2574 generic.go:356] "Generic (PLEG): container finished" podID="639b5027-5404-4d24-9b40-16faad8046fb" containerID="b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df" exitCode=0 Jul 24 17:49:50.928315 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:50.927898 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" event={"ID":"639b5027-5404-4d24-9b40-16faad8046fb","Type":"ContainerDied","Data":"b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df"} Jul 24 17:49:50.928360 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:50.928312 2574 scope.go:117] "RemoveContainer" containerID="b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df" Jul 24 17:49:50.929375 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:50.929341 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:49:50.929444 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:50.929429 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:49:50.929490 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:50.929475 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:49:51.843488 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:51.843385 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:51.937758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:51.937699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" event={"ID":"639b5027-5404-4d24-9b40-16faad8046fb","Type":"ContainerStarted","Data":"134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5"} Jul 24 17:49:52.406818 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:52.406783 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:52.408107 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:52.408082 2574 patch_prober.go:28] interesting pod/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 24 17:49:52.408222 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:52.408134 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 24 17:49:53.406630 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:53.406590 2574 patch_prober.go:28] interesting pod/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 24 17:49:53.407089 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:53.406651 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 24 17:49:54.407215 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:54.407174 2574 patch_prober.go:28] interesting pod/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 24 17:49:54.407639 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:54.407246 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 24 17:49:55.407067 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:55.407030 2574 patch_prober.go:28] interesting pod/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 24 17:49:55.407248 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:55.407086 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 24 17:49:56.410445 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:56.410417 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:56.410873 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:56.410803 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:49:56.411546 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:49:56.411526 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:50:02.121521 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.121484 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc"] Jul 24 17:50:02.121923 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.121909 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" Jul 24 17:50:02.121979 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.121924 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" Jul 24 17:50:02.122015 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.122006 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" Jul 24 17:50:02.125258 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.125241 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" Jul 24 17:50:02.128299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.128279 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:50:02.133519 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.132975 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc"] Jul 24 17:50:02.245993 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.245955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b648be-95e0-49c0-9b87-ebb4b5e851c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc\" (UID: \"51b648be-95e0-49c0-9b87-ebb4b5e851c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" Jul 24 17:50:02.346891 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.346856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b648be-95e0-49c0-9b87-ebb4b5e851c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc\" (UID: \"51b648be-95e0-49c0-9b87-ebb4b5e851c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" Jul 24 17:50:02.347536 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.347512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b648be-95e0-49c0-9b87-ebb4b5e851c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc\" (UID: \"51b648be-95e0-49c0-9b87-ebb4b5e851c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" Jul 24 17:50:02.437837 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.437758 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" Jul 24 17:50:02.573020 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:02.572996 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc"] Jul 24 17:50:02.574842 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:50:02.574811 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b648be_95e0_49c0_9b87_ebb4b5e851c5.slice/crio-914b7587bf1b40fcaecdc1429a79796bca744960012bfd3760efe6419ed0e03c WatchSource:0}: Error finding container 914b7587bf1b40fcaecdc1429a79796bca744960012bfd3760efe6419ed0e03c: Status 404 returned error can't find the container with id 914b7587bf1b40fcaecdc1429a79796bca744960012bfd3760efe6419ed0e03c Jul 24 17:50:03.014951 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:03.014856 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" event={"ID":"51b648be-95e0-49c0-9b87-ebb4b5e851c5","Type":"ContainerStarted","Data":"def52305869d8cca9e6380658be1a857790fc200862b1893fed2ed2de4fa71ff"} Jul 24 17:50:03.014951 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:03.014909 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" event={"ID":"51b648be-95e0-49c0-9b87-ebb4b5e851c5","Type":"ContainerStarted","Data":"914b7587bf1b40fcaecdc1429a79796bca744960012bfd3760efe6419ed0e03c"} Jul 24 17:50:03.034764 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:03.034697 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" podStartSLOduration=1.034682777 podStartE2EDuration="1.034682777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:03.032391106 +0000 UTC m=+1959.697965744" watchObservedRunningTime="2026-07-24 17:50:03.034682777 +0000 UTC m=+1959.700257413" Jul 24 17:50:04.021567 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:04.021532 2574 generic.go:356] "Generic (PLEG): container finished" podID="51b648be-95e0-49c0-9b87-ebb4b5e851c5" containerID="def52305869d8cca9e6380658be1a857790fc200862b1893fed2ed2de4fa71ff" exitCode=0 Jul 24 17:50:04.021964 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:04.021602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" event={"ID":"51b648be-95e0-49c0-9b87-ebb4b5e851c5","Type":"ContainerDied","Data":"def52305869d8cca9e6380658be1a857790fc200862b1893fed2ed2de4fa71ff"} Jul 24 17:50:05.163070 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:05.163048 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" Jul 24 17:50:05.273469 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:05.273435 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b648be-95e0-49c0-9b87-ebb4b5e851c5-maas-api-service-ca\") pod \"51b648be-95e0-49c0-9b87-ebb4b5e851c5\" (UID: \"51b648be-95e0-49c0-9b87-ebb4b5e851c5\") " Jul 24 17:50:05.273874 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:05.273851 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b648be-95e0-49c0-9b87-ebb4b5e851c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51b648be-95e0-49c0-9b87-ebb4b5e851c5" (UID: "51b648be-95e0-49c0-9b87-ebb4b5e851c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:50:05.375143 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:05.375055 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b648be-95e0-49c0-9b87-ebb4b5e851c5-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:06.033895 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:06.033867 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" Jul 24 17:50:06.034063 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:06.033907 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc" event={"ID":"51b648be-95e0-49c0-9b87-ebb4b5e851c5","Type":"ContainerDied","Data":"914b7587bf1b40fcaecdc1429a79796bca744960012bfd3760efe6419ed0e03c"} Jul 24 17:50:06.034063 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:06.033957 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="914b7587bf1b40fcaecdc1429a79796bca744960012bfd3760efe6419ed0e03c" Jul 24 17:50:16.065993 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:16.065903 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:50:17.232575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:17.232536 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc"] Jul 24 17:50:17.236134 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:17.236109 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7442c1c-4djgc"] Jul 24 17:50:17.402347 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:17.402318 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv"] Jul 24 17:50:17.402548 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:17.402524 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" containerID="cri-o://134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5" gracePeriod=30 Jul 24 17:50:17.801824 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:17.801791 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b648be-95e0-49c0-9b87-ebb4b5e851c5" path="/var/lib/kubelet/pods/51b648be-95e0-49c0-9b87-ebb4b5e851c5/volumes" Jul 24 17:50:22.669858 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.669794 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:50:22.723032 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.722994 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639b5027-5404-4d24-9b40-16faad8046fb-istio-podinfo\") pod \"639b5027-5404-4d24-9b40-16faad8046fb\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " Jul 24 17:50:22.723223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723045 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-certs\") pod \"639b5027-5404-4d24-9b40-16faad8046fb\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " Jul 24 17:50:22.723223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723072 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639b5027-5404-4d24-9b40-16faad8046fb-istiod-ca-cert\") pod \"639b5027-5404-4d24-9b40-16faad8046fb\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " Jul 24 17:50:22.723223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723124 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-istio-token\") pod \"639b5027-5404-4d24-9b40-16faad8046fb\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " Jul 24 17:50:22.723223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723154 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-socket\") pod \"639b5027-5404-4d24-9b40-16faad8046fb\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " Jul 24 17:50:22.723223 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723180 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-envoy\") pod \"639b5027-5404-4d24-9b40-16faad8046fb\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " Jul 24 17:50:22.723483 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723232 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-data\") pod \"639b5027-5404-4d24-9b40-16faad8046fb\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " Jul 24 17:50:22.723483 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723288 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-credential-socket\") pod \"639b5027-5404-4d24-9b40-16faad8046fb\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " Jul 24 17:50:22.723483 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723345 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz8jr\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-kube-api-access-rz8jr\") pod \"639b5027-5404-4d24-9b40-16faad8046fb\" (UID: \"639b5027-5404-4d24-9b40-16faad8046fb\") " Jul 24 17:50:22.723483 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723409 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "639b5027-5404-4d24-9b40-16faad8046fb" (UID: "639b5027-5404-4d24-9b40-16faad8046fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:50:22.723691 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723490 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639b5027-5404-4d24-9b40-16faad8046fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "639b5027-5404-4d24-9b40-16faad8046fb" (UID: "639b5027-5404-4d24-9b40-16faad8046fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:50:22.723691 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723659 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "639b5027-5404-4d24-9b40-16faad8046fb" (UID: "639b5027-5404-4d24-9b40-16faad8046fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:50:22.723853 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723709 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "639b5027-5404-4d24-9b40-16faad8046fb" (UID: "639b5027-5404-4d24-9b40-16faad8046fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:50:22.723853 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723805 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:22.723853 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723824 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639b5027-5404-4d24-9b40-16faad8046fb-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:22.723853 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723838 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:22.723853 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.723850 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:22.724326 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.724302 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "639b5027-5404-4d24-9b40-16faad8046fb" (UID: "639b5027-5404-4d24-9b40-16faad8046fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:50:22.725370 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.725341 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/639b5027-5404-4d24-9b40-16faad8046fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "639b5027-5404-4d24-9b40-16faad8046fb" (UID: "639b5027-5404-4d24-9b40-16faad8046fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:50:22.725631 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.725609 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "639b5027-5404-4d24-9b40-16faad8046fb" (UID: "639b5027-5404-4d24-9b40-16faad8046fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:50:22.725950 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.725924 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "639b5027-5404-4d24-9b40-16faad8046fb" (UID: "639b5027-5404-4d24-9b40-16faad8046fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:50:22.725950 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.725935 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-kube-api-access-rz8jr" (OuterVolumeSpecName: "kube-api-access-rz8jr") pod "639b5027-5404-4d24-9b40-16faad8046fb" (UID: "639b5027-5404-4d24-9b40-16faad8046fb"). InnerVolumeSpecName "kube-api-access-rz8jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:50:22.825112 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.825077 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz8jr\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-kube-api-access-rz8jr\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:22.825112 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.825104 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639b5027-5404-4d24-9b40-16faad8046fb-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:22.825112 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.825113 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639b5027-5404-4d24-9b40-16faad8046fb-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:22.825112 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.825121 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:22.825380 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:22.825130 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639b5027-5404-4d24-9b40-16faad8046fb-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:50:23.139650 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.139615 2574 generic.go:356] "Generic (PLEG): container finished" podID="639b5027-5404-4d24-9b40-16faad8046fb" containerID="134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5" exitCode=0 Jul 24 17:50:23.139843 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.139669 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" Jul 24 17:50:23.139843 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.139703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" event={"ID":"639b5027-5404-4d24-9b40-16faad8046fb","Type":"ContainerDied","Data":"134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5"} Jul 24 17:50:23.139843 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.139794 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv" event={"ID":"639b5027-5404-4d24-9b40-16faad8046fb","Type":"ContainerDied","Data":"806cc145aa4ac512c5ef43b3a98b2d5cf610d7b79aabd72aa40b68dfa28d2822"} Jul 24 17:50:23.139843 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.139830 2574 scope.go:117] "RemoveContainer" containerID="134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5" Jul 24 17:50:23.149905 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.149886 2574 scope.go:117] "RemoveContainer" containerID="b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df" Jul 24 17:50:23.158987 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.158967 2574 scope.go:117] "RemoveContainer" containerID="134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5" Jul 24 17:50:23.159254 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:50:23.159235 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5\": container with ID starting with 134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5 not found: ID does not exist" containerID="134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5" Jul 24 17:50:23.159294 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.159265 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5"} err="failed to get container status \"134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5\": rpc error: code = NotFound desc = could not find container \"134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5\": container with ID starting with 134815000cfb3720176a4e518286659a82cc8f40832b2dc775155447cca294c5 not found: ID does not exist" Jul 24 17:50:23.159294 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.159284 2574 scope.go:117] "RemoveContainer" containerID="b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df" Jul 24 17:50:23.159519 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:50:23.159501 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df\": container with ID starting with b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df not found: ID does not exist" containerID="b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df" Jul 24 17:50:23.159565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.159528 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df"} err="failed to get container status \"b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df\": rpc error: code = NotFound desc = could not find container \"b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df\": container with ID starting with b0c42542ed9f77ccdda7cf775b431646c127e3bcc79b194d32e1d0e43d58b1df not found: ID does not exist" Jul 24 17:50:23.165090 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.165066 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv"] Jul 24 17:50:23.171978 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.171955 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e7442c1c-openshift-default-5ddb6657bd-t2tlv"] Jul 24 17:50:23.802253 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:50:23.802223 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="639b5027-5404-4d24-9b40-16faad8046fb" path="/var/lib/kubelet/pods/639b5027-5404-4d24-9b40-16faad8046fb/volumes" Jul 24 17:51:48.627038 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:51:48.627004 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:51:58.105762 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:51:58.105711 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:52:04.656653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.656618 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c"] Jul 24 17:52:04.657130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657040 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" Jul 24 17:52:04.657130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657052 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" Jul 24 17:52:04.657130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657062 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" Jul 24 17:52:04.657130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657068 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" Jul 24 17:52:04.657130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657085 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" Jul 24 17:52:04.657130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657090 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" Jul 24 17:52:04.657130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657098 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b648be-95e0-49c0-9b87-ebb4b5e851c5" containerName="revoke-keys" Jul 24 17:52:04.657130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657103 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b648be-95e0-49c0-9b87-ebb4b5e851c5" containerName="revoke-keys" Jul 24 17:52:04.657385 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657166 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" Jul 24 17:52:04.657385 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657173 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d13fe6e-7ec1-45b8-aa24-86408f8a2cec" containerName="istio-proxy" Jul 24 17:52:04.657385 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.657179 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b648be-95e0-49c0-9b87-ebb4b5e851c5" containerName="revoke-keys" Jul 24 17:52:04.660257 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.660238 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.664264 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.664245 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-117b3f04-openshift-default-dockercfg-qxpwj\"" Jul 24 17:52:04.699797 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.699766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c"] Jul 24 17:52:04.757407 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.757376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-certs\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.757588 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.757428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-socket\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.757588 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.757450 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-token\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.757588 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.757475 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrbfl\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-kube-api-access-jrbfl\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.757588 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.757529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-data\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.757588 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.757551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-envoy\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.757588 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.757570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-credential-socket\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.757588 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.757586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-podinfo\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.757869 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.757610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istiod-ca-cert\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858321 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-data\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858321 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-envoy\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858549 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-credential-socket\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858549 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-podinfo\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858549 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istiod-ca-cert\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858549 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858516 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-certs\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858791 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858573 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-socket\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858791 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-token\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858791 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrbfl\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-kube-api-access-jrbfl\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.858791 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-data\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.859002 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858849 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-credential-socket\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.859002 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.858914 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-certs\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.859146 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.859123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-socket\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.859493 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.859469 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istiod-ca-cert\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.860899 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.860880 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-envoy\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.861147 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.861127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-podinfo\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.868886 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.868864 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-token\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.869008 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.868985 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrbfl\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-kube-api-access-jrbfl\") pod \"e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:04.970817 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:04.970784 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:05.117171 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:05.117127 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c"] Jul 24 17:52:05.119593 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:52:05.119571 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c4efc8_99e2_4d98_904f_ece07bac95f8.slice/crio-cc52c99c484162a771c719ca93d0e7d54a3b56f743b79754bce40fe2208cf032 WatchSource:0}: Error finding container cc52c99c484162a771c719ca93d0e7d54a3b56f743b79754bce40fe2208cf032: Status 404 returned error can't find the container with id cc52c99c484162a771c719ca93d0e7d54a3b56f743b79754bce40fe2208cf032 Jul 24 17:52:05.121759 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:05.121709 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:52:05.121872 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:05.121810 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:52:05.121872 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:05.121855 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:52:05.743523 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:05.743482 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" event={"ID":"c6c4efc8-99e2-4d98-904f-ece07bac95f8","Type":"ContainerStarted","Data":"248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855"} Jul 24 17:52:05.743948 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:05.743536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" event={"ID":"c6c4efc8-99e2-4d98-904f-ece07bac95f8","Type":"ContainerStarted","Data":"cc52c99c484162a771c719ca93d0e7d54a3b56f743b79754bce40fe2208cf032"} Jul 24 17:52:05.774704 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:05.774652 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" podStartSLOduration=1.774636145 podStartE2EDuration="1.774636145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:05.773537253 +0000 UTC m=+2082.439111891" watchObservedRunningTime="2026-07-24 17:52:05.774636145 +0000 UTC m=+2082.440210782" Jul 24 17:52:05.971439 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:05.971401 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:06.976905 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:06.976873 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:07.755475 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:07.755440 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:07.756410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:07.756389 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:12.331958 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:12.331922 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:52:20.834412 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:20.834378 2574 generic.go:356] "Generic (PLEG): container finished" podID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerID="248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855" exitCode=0 Jul 24 17:52:20.834846 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:20.834449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" event={"ID":"c6c4efc8-99e2-4d98-904f-ece07bac95f8","Type":"ContainerDied","Data":"248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855"} Jul 24 17:52:20.834892 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:20.834856 2574 scope.go:117] "RemoveContainer" containerID="248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855" Jul 24 17:52:20.835915 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:20.835888 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:52:20.835978 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:20.835956 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:52:20.836014 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:20.835991 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:52:21.049924 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.049882 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg"] Jul 24 17:52:21.050611 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.050589 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="639b5027-5404-4d24-9b40-16faad8046fb" containerName="istio-proxy" Jul 24 17:52:21.053187 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.053168 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" Jul 24 17:52:21.057138 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.057116 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:52:21.122216 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.122129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe1ad0-34cc-4e42-b74f-f348e4e07163-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-117b3f04-72zzg\" (UID: \"73fe1ad0-34cc-4e42-b74f-f348e4e07163\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" Jul 24 17:52:21.132835 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.132798 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg"] Jul 24 17:52:21.222843 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.222806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe1ad0-34cc-4e42-b74f-f348e4e07163-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-117b3f04-72zzg\" (UID: \"73fe1ad0-34cc-4e42-b74f-f348e4e07163\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" Jul 24 17:52:21.223427 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.223404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe1ad0-34cc-4e42-b74f-f348e4e07163-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-117b3f04-72zzg\" (UID: \"73fe1ad0-34cc-4e42-b74f-f348e4e07163\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" Jul 24 17:52:21.363854 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.363797 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" Jul 24 17:52:21.514409 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.514371 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg"] Jul 24 17:52:21.517703 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:52:21.517679 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fe1ad0_34cc_4e42_b74f_f348e4e07163.slice/crio-14dd222fc218e74a12902685bb2ddbfc03ef1f1957e4596717c80c83da277f26 WatchSource:0}: Error finding container 14dd222fc218e74a12902685bb2ddbfc03ef1f1957e4596717c80c83da277f26: Status 404 returned error can't find the container with id 14dd222fc218e74a12902685bb2ddbfc03ef1f1957e4596717c80c83da277f26 Jul 24 17:52:21.845623 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.845572 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" event={"ID":"c6c4efc8-99e2-4d98-904f-ece07bac95f8","Type":"ContainerStarted","Data":"b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c"} Jul 24 17:52:21.849404 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.849366 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" event={"ID":"73fe1ad0-34cc-4e42-b74f-f348e4e07163","Type":"ContainerStarted","Data":"14dd222fc218e74a12902685bb2ddbfc03ef1f1957e4596717c80c83da277f26"} Jul 24 17:52:21.971831 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.971780 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:21.973107 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.973082 2574 patch_prober.go:28] interesting pod/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 24 17:52:21.973249 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:21.973133 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 24 17:52:22.756317 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:22.756264 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:22.860044 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:22.859947 2574 generic.go:356] "Generic (PLEG): container finished" podID="73fe1ad0-34cc-4e42-b74f-f348e4e07163" containerID="fc92a7e3710a80c2115795f57b40d10f41c2a24db448646c1703cb279268f4c1" exitCode=0 Jul 24 17:52:22.860515 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:22.860036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" event={"ID":"73fe1ad0-34cc-4e42-b74f-f348e4e07163","Type":"ContainerDied","Data":"fc92a7e3710a80c2115795f57b40d10f41c2a24db448646c1703cb279268f4c1"} Jul 24 17:52:22.971484 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:22.971449 2574 patch_prober.go:28] interesting pod/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 24 17:52:22.971657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:22.971504 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 24 17:52:23.927887 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:23.927859 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:52:23.931783 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:23.931763 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:52:23.971244 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:23.971217 2574 patch_prober.go:28] interesting pod/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 24 17:52:23.971486 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:23.971458 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 24 17:52:24.024485 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:24.024463 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" Jul 24 17:52:24.166937 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:24.166824 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe1ad0-34cc-4e42-b74f-f348e4e07163-maas-api-service-ca\") pod \"73fe1ad0-34cc-4e42-b74f-f348e4e07163\" (UID: \"73fe1ad0-34cc-4e42-b74f-f348e4e07163\") " Jul 24 17:52:24.167180 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:24.167154 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fe1ad0-34cc-4e42-b74f-f348e4e07163-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73fe1ad0-34cc-4e42-b74f-f348e4e07163" (UID: "73fe1ad0-34cc-4e42-b74f-f348e4e07163"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:52:24.268281 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:24.268244 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fe1ad0-34cc-4e42-b74f-f348e4e07163-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:24.875687 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:24.875653 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" Jul 24 17:52:24.875907 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:24.875686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg" event={"ID":"73fe1ad0-34cc-4e42-b74f-f348e4e07163","Type":"ContainerDied","Data":"14dd222fc218e74a12902685bb2ddbfc03ef1f1957e4596717c80c83da277f26"} Jul 24 17:52:24.875907 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:24.875747 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14dd222fc218e74a12902685bb2ddbfc03ef1f1957e4596717c80c83da277f26" Jul 24 17:52:24.972026 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:24.971982 2574 patch_prober.go:28] interesting pod/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 24 17:52:24.972418 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:24.972067 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 24 17:52:25.975407 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:25.975378 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:25.976414 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:25.976393 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:34.477756 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:34.477692 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:52:36.204002 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:36.203970 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg"] Jul 24 17:52:36.206262 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:36.206238 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-117b3f04-72zzg"] Jul 24 17:52:37.801481 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:37.801452 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73fe1ad0-34cc-4e42-b74f-f348e4e07163" path="/var/lib/kubelet/pods/73fe1ad0-34cc-4e42-b74f-f348e4e07163/volumes" Jul 24 17:52:42.911064 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:42.911026 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c"] Jul 24 17:52:42.911490 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:42.911280 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" containerID="cri-o://b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c" gracePeriod=30 Jul 24 17:52:48.177654 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.177585 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:48.273424 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273393 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-token\") pod \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " Jul 24 17:52:48.273605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273432 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istiod-ca-cert\") pod \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " Jul 24 17:52:48.273605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273474 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-envoy\") pod \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " Jul 24 17:52:48.273605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273514 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-certs\") pod \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " Jul 24 17:52:48.273605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273541 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrbfl\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-kube-api-access-jrbfl\") pod \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " Jul 24 17:52:48.273852 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273648 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-socket\") pod \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " Jul 24 17:52:48.273852 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273676 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-data\") pod \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " Jul 24 17:52:48.273852 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273717 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-credential-socket\") pod \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " Jul 24 17:52:48.273852 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273815 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-podinfo\") pod \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\" (UID: \"c6c4efc8-99e2-4d98-904f-ece07bac95f8\") " Jul 24 17:52:48.274058 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.273887 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6c4efc8-99e2-4d98-904f-ece07bac95f8" (UID: "c6c4efc8-99e2-4d98-904f-ece07bac95f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:48.274058 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.274033 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6c4efc8-99e2-4d98-904f-ece07bac95f8" (UID: "c6c4efc8-99e2-4d98-904f-ece07bac95f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:52:48.274183 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.274065 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6c4efc8-99e2-4d98-904f-ece07bac95f8" (UID: "c6c4efc8-99e2-4d98-904f-ece07bac95f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:48.274248 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.274206 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:48.274248 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.274227 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:48.274357 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.274241 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:48.274663 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.274616 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6c4efc8-99e2-4d98-904f-ece07bac95f8" (UID: "c6c4efc8-99e2-4d98-904f-ece07bac95f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:48.275210 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.275186 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6c4efc8-99e2-4d98-904f-ece07bac95f8" (UID: "c6c4efc8-99e2-4d98-904f-ece07bac95f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:48.276029 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.276003 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6c4efc8-99e2-4d98-904f-ece07bac95f8" (UID: "c6c4efc8-99e2-4d98-904f-ece07bac95f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:52:48.276122 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.276098 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6c4efc8-99e2-4d98-904f-ece07bac95f8" (UID: "c6c4efc8-99e2-4d98-904f-ece07bac95f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:48.276298 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.276266 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6c4efc8-99e2-4d98-904f-ece07bac95f8" (UID: "c6c4efc8-99e2-4d98-904f-ece07bac95f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:52:48.276494 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.276475 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-kube-api-access-jrbfl" (OuterVolumeSpecName: "kube-api-access-jrbfl") pod "c6c4efc8-99e2-4d98-904f-ece07bac95f8" (UID: "c6c4efc8-99e2-4d98-904f-ece07bac95f8"). InnerVolumeSpecName "kube-api-access-jrbfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:52:48.375497 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.375463 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:48.375497 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.375493 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:48.375497 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.375504 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:48.375773 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.375512 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:48.375773 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.375521 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c4efc8-99e2-4d98-904f-ece07bac95f8-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:48.375773 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:48.375529 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrbfl\" (UniqueName: \"kubernetes.io/projected/c6c4efc8-99e2-4d98-904f-ece07bac95f8-kube-api-access-jrbfl\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:52:49.032067 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.032035 2574 generic.go:356] "Generic (PLEG): container finished" podID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerID="b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c" exitCode=0 Jul 24 17:52:49.032256 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.032111 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" Jul 24 17:52:49.032256 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.032128 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" event={"ID":"c6c4efc8-99e2-4d98-904f-ece07bac95f8","Type":"ContainerDied","Data":"b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c"} Jul 24 17:52:49.032256 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.032186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c" event={"ID":"c6c4efc8-99e2-4d98-904f-ece07bac95f8","Type":"ContainerDied","Data":"cc52c99c484162a771c719ca93d0e7d54a3b56f743b79754bce40fe2208cf032"} Jul 24 17:52:49.032256 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.032213 2574 scope.go:117] "RemoveContainer" containerID="b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c" Jul 24 17:52:49.042696 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.042675 2574 scope.go:117] "RemoveContainer" containerID="248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855" Jul 24 17:52:49.051525 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.051504 2574 scope.go:117] "RemoveContainer" containerID="b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c" Jul 24 17:52:49.051824 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:52:49.051804 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c\": container with ID starting with b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c not found: ID does not exist" containerID="b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c" Jul 24 17:52:49.051918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.051837 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c"} err="failed to get container status \"b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c\": rpc error: code = NotFound desc = could not find container \"b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c\": container with ID starting with b0c770d447ee72233150ca961319ba0446a0f207ad98452fb26ad11c2d0b896c not found: ID does not exist" Jul 24 17:52:49.051918 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.051864 2574 scope.go:117] "RemoveContainer" containerID="248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855" Jul 24 17:52:49.052114 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:52:49.052094 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855\": container with ID starting with 248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855 not found: ID does not exist" containerID="248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855" Jul 24 17:52:49.052164 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.052122 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855"} err="failed to get container status \"248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855\": rpc error: code = NotFound desc = could not find container \"248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855\": container with ID starting with 248a72e57c80c10af4e431d7fcf662a76123bd0f2af696eb0dd502609f7c6855 not found: ID does not exist" Jul 24 17:52:49.071581 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.071544 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c"] Jul 24 17:52:49.078231 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.078202 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-117b3f04-openshift-default-7d867fc4fb-pxf7c"] Jul 24 17:52:49.801181 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:52:49.801147 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" path="/var/lib/kubelet/pods/c6c4efc8-99e2-4d98-904f-ece07bac95f8/volumes" Jul 24 17:53:04.173009 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.172974 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg"] Jul 24 17:53:04.173565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.173545 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73fe1ad0-34cc-4e42-b74f-f348e4e07163" containerName="revoke-keys" Jul 24 17:53:04.173645 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.173567 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="73fe1ad0-34cc-4e42-b74f-f348e4e07163" containerName="revoke-keys" Jul 24 17:53:04.173645 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.173585 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" Jul 24 17:53:04.173645 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.173594 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" Jul 24 17:53:04.173645 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.173610 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" Jul 24 17:53:04.173645 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.173618 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" Jul 24 17:53:04.173914 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.173739 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" Jul 24 17:53:04.173914 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.173753 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c4efc8-99e2-4d98-904f-ece07bac95f8" containerName="istio-proxy" Jul 24 17:53:04.173914 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.173766 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="73fe1ad0-34cc-4e42-b74f-f348e4e07163" containerName="revoke-keys" Jul 24 17:53:04.176944 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.176924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.180841 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.180821 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9ae4eef5-openshift-default-dockercfg-x7xrp\"" Jul 24 17:53:04.212573 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.212548 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg"] Jul 24 17:53:04.316534 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.316496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-certs\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.316534 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.316537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-data\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.316766 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.316560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-token\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.316766 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.316587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-credential-socket\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.316766 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.316646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlk9v\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-kube-api-access-qlk9v\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.316766 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.316718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-podinfo\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.316766 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.316762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istiod-ca-cert\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.317254 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.316801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-envoy\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.317254 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.316840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-socket\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.417658 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.417620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-socket\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.417895 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.417713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-certs\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.417895 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.417783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-data\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.417895 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.417820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-token\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.417895 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.417855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-credential-socket\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.418137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.417899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlk9v\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-kube-api-access-qlk9v\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.418137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.418056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-podinfo\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.418137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.418108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istiod-ca-cert\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.418137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.418130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-certs\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.418321 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.418144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-envoy\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.418321 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.418235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-socket\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.418426 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.418413 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-data\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.418498 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.418474 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-credential-socket\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.418799 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.418778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istiod-ca-cert\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.420605 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.420563 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-envoy\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.420715 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.420699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-podinfo\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.427627 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.427575 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-token\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.428085 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.428062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlk9v\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-kube-api-access-qlk9v\") pod \"e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.488793 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.488766 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:04.634417 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.634387 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg"] Jul 24 17:53:04.636267 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:53:04.636240 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb38ea4_89dd_4856_be8c_3dc592659b5d.slice/crio-de80a13e4af49bf5fbeb1a0209e7ef8aa45427bf4f9940a076577c74238c26be WatchSource:0}: Error finding container de80a13e4af49bf5fbeb1a0209e7ef8aa45427bf4f9940a076577c74238c26be: Status 404 returned error can't find the container with id de80a13e4af49bf5fbeb1a0209e7ef8aa45427bf4f9940a076577c74238c26be Jul 24 17:53:04.638462 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.638432 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:53:04.638561 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.638492 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:53:04.638561 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:04.638523 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:53:05.131147 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:05.131113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" event={"ID":"7bb38ea4-89dd-4856-be8c-3dc592659b5d","Type":"ContainerStarted","Data":"cd560a58273ba7efe2566ba0a9f8dabc49deb5e51aab8e3622fefe24d0ea4978"} Jul 24 17:53:05.131329 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:05.131172 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" event={"ID":"7bb38ea4-89dd-4856-be8c-3dc592659b5d","Type":"ContainerStarted","Data":"de80a13e4af49bf5fbeb1a0209e7ef8aa45427bf4f9940a076577c74238c26be"} Jul 24 17:53:05.156262 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:05.156214 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" podStartSLOduration=1.156196229 podStartE2EDuration="1.156196229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:53:05.154128556 +0000 UTC m=+2141.819703193" watchObservedRunningTime="2026-07-24 17:53:05.156196229 +0000 UTC m=+2141.821770937" Jul 24 17:53:05.489087 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:05.489053 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:05.493600 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:05.493577 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:06.137215 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:06.137183 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:06.138220 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:06.138198 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:12.410952 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:12.410915 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:53:20.229251 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:20.229215 2574 generic.go:356] "Generic (PLEG): container finished" podID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerID="cd560a58273ba7efe2566ba0a9f8dabc49deb5e51aab8e3622fefe24d0ea4978" exitCode=0 Jul 24 17:53:20.229681 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:20.229294 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" event={"ID":"7bb38ea4-89dd-4856-be8c-3dc592659b5d","Type":"ContainerDied","Data":"cd560a58273ba7efe2566ba0a9f8dabc49deb5e51aab8e3622fefe24d0ea4978"} Jul 24 17:53:20.229681 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:20.229672 2574 scope.go:117] "RemoveContainer" containerID="cd560a58273ba7efe2566ba0a9f8dabc49deb5e51aab8e3622fefe24d0ea4978" Jul 24 17:53:20.230673 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:20.230642 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:53:20.230799 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:20.230744 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:53:20.230872 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:20.230799 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:53:21.137562 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:21.137527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:21.239427 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:21.239377 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" event={"ID":"7bb38ea4-89dd-4856-be8c-3dc592659b5d","Type":"ContainerStarted","Data":"7626a6c61c15ef5dd2a5438e24f617fb8c7b1899eeacb373b751761cc3a7a32f"} Jul 24 17:53:21.489189 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:21.489150 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:21.490301 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:21.490274 2574 patch_prober.go:28] interesting pod/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 24 17:53:21.490445 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:21.490330 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 24 17:53:22.489641 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:22.489602 2574 patch_prober.go:28] interesting pod/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 24 17:53:22.490093 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:22.489659 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 24 17:53:23.489999 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:23.489964 2574 patch_prober.go:28] interesting pod/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 24 17:53:23.490435 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:23.490015 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 24 17:53:24.489523 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:24.489487 2574 patch_prober.go:28] interesting pod/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 24 17:53:24.489763 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:24.489554 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 24 17:53:25.493634 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:25.493596 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:25.494042 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:25.493979 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:25.494618 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:25.494602 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:53:30.578608 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:30.578571 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:53:52.178417 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:52.178380 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:53:56.739518 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:56.739482 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk"] Jul 24 17:53:56.742642 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:56.742625 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" Jul 24 17:53:56.747670 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:56.747649 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:53:56.751844 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:56.751819 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk"] Jul 24 17:53:56.796091 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:56.796054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b9152c-0b9f-4927-a43c-27e34aee1a88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk\" (UID: \"80b9152c-0b9f-4927-a43c-27e34aee1a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" Jul 24 17:53:56.897506 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:56.897467 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b9152c-0b9f-4927-a43c-27e34aee1a88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk\" (UID: \"80b9152c-0b9f-4927-a43c-27e34aee1a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" Jul 24 17:53:56.898147 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:56.898125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b9152c-0b9f-4927-a43c-27e34aee1a88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk\" (UID: \"80b9152c-0b9f-4927-a43c-27e34aee1a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" Jul 24 17:53:57.055113 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:57.055022 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" Jul 24 17:53:57.208124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:57.208085 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk"] Jul 24 17:53:57.208912 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:53:57.208887 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b9152c_0b9f_4927_a43c_27e34aee1a88.slice/crio-07afb8b5e6990cb2c241e62c91fd0fdf8bfdb847869774016561524ac62e3068 WatchSource:0}: Error finding container 07afb8b5e6990cb2c241e62c91fd0fdf8bfdb847869774016561524ac62e3068: Status 404 returned error can't find the container with id 07afb8b5e6990cb2c241e62c91fd0fdf8bfdb847869774016561524ac62e3068 Jul 24 17:53:57.460027 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:57.459996 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" event={"ID":"80b9152c-0b9f-4927-a43c-27e34aee1a88","Type":"ContainerStarted","Data":"07afb8b5e6990cb2c241e62c91fd0fdf8bfdb847869774016561524ac62e3068"} Jul 24 17:53:58.467483 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:58.467446 2574 generic.go:356] "Generic (PLEG): container finished" podID="80b9152c-0b9f-4927-a43c-27e34aee1a88" containerID="9b6d29852cca106b022b0b782ae84ba309a4449220aae185460e9c84f26a684f" exitCode=0 Jul 24 17:53:58.467997 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:58.467494 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" event={"ID":"80b9152c-0b9f-4927-a43c-27e34aee1a88","Type":"ContainerDied","Data":"9b6d29852cca106b022b0b782ae84ba309a4449220aae185460e9c84f26a684f"} Jul 24 17:53:59.610919 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:59.610888 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" Jul 24 17:53:59.722479 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:59.722443 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b9152c-0b9f-4927-a43c-27e34aee1a88-maas-api-service-ca\") pod \"80b9152c-0b9f-4927-a43c-27e34aee1a88\" (UID: \"80b9152c-0b9f-4927-a43c-27e34aee1a88\") " Jul 24 17:53:59.722848 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:59.722828 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b9152c-0b9f-4927-a43c-27e34aee1a88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80b9152c-0b9f-4927-a43c-27e34aee1a88" (UID: "80b9152c-0b9f-4927-a43c-27e34aee1a88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:53:59.824006 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:53:59.823976 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b9152c-0b9f-4927-a43c-27e34aee1a88-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:54:00.480106 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:54:00.480073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" event={"ID":"80b9152c-0b9f-4927-a43c-27e34aee1a88","Type":"ContainerDied","Data":"07afb8b5e6990cb2c241e62c91fd0fdf8bfdb847869774016561524ac62e3068"} Jul 24 17:54:00.480313 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:54:00.480135 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07afb8b5e6990cb2c241e62c91fd0fdf8bfdb847869774016561524ac62e3068" Jul 24 17:54:00.480313 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:54:00.480083 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk" Jul 24 17:54:10.966924 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:54:10.966890 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:54:16.847513 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:54:16.847470 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk"] Jul 24 17:54:16.852271 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:54:16.852242 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9ae4eef5-hn2wk"] Jul 24 17:54:17.801996 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:54:17.801962 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80b9152c-0b9f-4927-a43c-27e34aee1a88" path="/var/lib/kubelet/pods/80b9152c-0b9f-4927-a43c-27e34aee1a88/volumes" Jul 24 17:54:19.678271 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:54:19.678233 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:54:24.035201 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:54:24.035172 2574 scope.go:117] "RemoveContainer" containerID="a7b76a8e7af6b089c4f43987b80ad611cdce0d97f8a39e73bbca9a616a56a44e" Jul 24 17:55:24.059945 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:24.059905 2574 scope.go:117] "RemoveContainer" containerID="2efa99a376e427c4bfb8cfeb76520dedc8105b40cd506ba89395c94ce9e179ca" Jul 24 17:55:24.069027 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:24.069008 2574 scope.go:117] "RemoveContainer" containerID="354ecead28f54ac1486fd9c6673fe9ffec90ab944aa47d4556969122d646bae7" Jul 24 17:55:27.883949 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:27.883914 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg"] Jul 24 17:55:27.884371 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:27.884129 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" containerID="cri-o://7626a6c61c15ef5dd2a5438e24f617fb8c7b1899eeacb373b751761cc3a7a32f" gracePeriod=30 Jul 24 17:55:31.984992 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:31.984943 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:55:33.042018 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.041987 2574 generic.go:356] "Generic (PLEG): container finished" podID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerID="7626a6c61c15ef5dd2a5438e24f617fb8c7b1899eeacb373b751761cc3a7a32f" exitCode=0 Jul 24 17:55:33.042414 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.042055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" event={"ID":"7bb38ea4-89dd-4856-be8c-3dc592659b5d","Type":"ContainerDied","Data":"7626a6c61c15ef5dd2a5438e24f617fb8c7b1899eeacb373b751761cc3a7a32f"} Jul 24 17:55:33.042414 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.042127 2574 scope.go:117] "RemoveContainer" containerID="cd560a58273ba7efe2566ba0a9f8dabc49deb5e51aab8e3622fefe24d0ea4978" Jul 24 17:55:33.141333 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.141309 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:55:33.185379 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185293 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-podinfo\") pod \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " Jul 24 17:55:33.185379 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185353 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-socket\") pod \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " Jul 24 17:55:33.185625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185414 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istiod-ca-cert\") pod \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " Jul 24 17:55:33.185625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185478 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-certs\") pod \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " Jul 24 17:55:33.185625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185505 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-data\") pod \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " Jul 24 17:55:33.185625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185530 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-credential-socket\") pod \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " Jul 24 17:55:33.185625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185563 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-envoy\") pod \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " Jul 24 17:55:33.185625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185603 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-token\") pod \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " Jul 24 17:55:33.185970 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185648 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlk9v\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-kube-api-access-qlk9v\") pod \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\" (UID: \"7bb38ea4-89dd-4856-be8c-3dc592659b5d\") " Jul 24 17:55:33.185970 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185652 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bb38ea4-89dd-4856-be8c-3dc592659b5d" (UID: "7bb38ea4-89dd-4856-be8c-3dc592659b5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:55:33.185970 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.185765 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bb38ea4-89dd-4856-be8c-3dc592659b5d" (UID: "7bb38ea4-89dd-4856-be8c-3dc592659b5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:55:33.186299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.186020 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:55:33.186299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.186040 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:55:33.186299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.186127 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bb38ea4-89dd-4856-be8c-3dc592659b5d" (UID: "7bb38ea4-89dd-4856-be8c-3dc592659b5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:55:33.186299 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.186194 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bb38ea4-89dd-4856-be8c-3dc592659b5d" (UID: "7bb38ea4-89dd-4856-be8c-3dc592659b5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:55:33.186978 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.186922 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bb38ea4-89dd-4856-be8c-3dc592659b5d" (UID: "7bb38ea4-89dd-4856-be8c-3dc592659b5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:55:33.187867 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.187841 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bb38ea4-89dd-4856-be8c-3dc592659b5d" (UID: "7bb38ea4-89dd-4856-be8c-3dc592659b5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:55:33.188265 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.188237 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bb38ea4-89dd-4856-be8c-3dc592659b5d" (UID: "7bb38ea4-89dd-4856-be8c-3dc592659b5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:55:33.188427 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.188399 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bb38ea4-89dd-4856-be8c-3dc592659b5d" (UID: "7bb38ea4-89dd-4856-be8c-3dc592659b5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:55:33.188427 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.188413 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-kube-api-access-qlk9v" (OuterVolumeSpecName: "kube-api-access-qlk9v") pod "7bb38ea4-89dd-4856-be8c-3dc592659b5d" (UID: "7bb38ea4-89dd-4856-be8c-3dc592659b5d"). InnerVolumeSpecName "kube-api-access-qlk9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:55:33.286897 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.286862 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:55:33.286897 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.286890 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:55:33.286897 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.286900 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:55:33.286897 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.286909 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:55:33.287175 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.286920 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:55:33.287175 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.286928 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlk9v\" (UniqueName: \"kubernetes.io/projected/7bb38ea4-89dd-4856-be8c-3dc592659b5d-kube-api-access-qlk9v\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:55:33.287175 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:33.286936 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb38ea4-89dd-4856-be8c-3dc592659b5d-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:55:34.055860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:34.055826 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" Jul 24 17:55:34.055860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:34.055835 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg" event={"ID":"7bb38ea4-89dd-4856-be8c-3dc592659b5d","Type":"ContainerDied","Data":"de80a13e4af49bf5fbeb1a0209e7ef8aa45427bf4f9940a076577c74238c26be"} Jul 24 17:55:34.056379 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:34.055915 2574 scope.go:117] "RemoveContainer" containerID="7626a6c61c15ef5dd2a5438e24f617fb8c7b1899eeacb373b751761cc3a7a32f" Jul 24 17:55:34.078093 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:34.078058 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg"] Jul 24 17:55:34.082374 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:34.082346 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9ae4eef5-openshift-default-648c6847bd-9kzdg"] Jul 24 17:55:35.801441 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:35.801414 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" path="/var/lib/kubelet/pods/7bb38ea4-89dd-4856-be8c-3dc592659b5d/volumes" Jul 24 17:55:41.770873 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:55:41.770837 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:56:24.095904 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.095870 2574 scope.go:117] "RemoveContainer" containerID="def52305869d8cca9e6380658be1a857790fc200862b1893fed2ed2de4fa71ff" Jul 24 17:56:24.971005 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.970972 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4"] Jul 24 17:56:24.971527 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.971513 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b9152c-0b9f-4927-a43c-27e34aee1a88" containerName="revoke-keys" Jul 24 17:56:24.971575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.971531 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b9152c-0b9f-4927-a43c-27e34aee1a88" containerName="revoke-keys" Jul 24 17:56:24.971575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.971547 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" Jul 24 17:56:24.971575 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.971556 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" Jul 24 17:56:24.971671 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.971576 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" Jul 24 17:56:24.971671 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.971584 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" Jul 24 17:56:24.971774 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.971688 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" Jul 24 17:56:24.971774 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.971699 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb38ea4-89dd-4856-be8c-3dc592659b5d" containerName="istio-proxy" Jul 24 17:56:24.971774 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.971709 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b9152c-0b9f-4927-a43c-27e34aee1a88" containerName="revoke-keys" Jul 24 17:56:24.976535 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.976516 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:24.980356 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.980336 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cdf40f-openshift-default-dockercfg-74tg5\"" Jul 24 17:56:24.997429 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:24.997408 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4"] Jul 24 17:56:25.052634 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.052587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-socket\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.052815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.052739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-podinfo\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.052815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.052778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-token\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.052815 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.052800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istiod-ca-cert\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.052938 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.052906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tfc\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-kube-api-access-f7tfc\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.052987 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.052956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-certs\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.053025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.052992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-data\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.053066 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.053034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-credential-socket\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.053066 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.053060 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-envoy\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154224 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tfc\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-kube-api-access-f7tfc\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-certs\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-data\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-credential-socket\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-envoy\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-socket\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154582 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-podinfo\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-token\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154624 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-data\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istiod-ca-cert\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.154653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-certs\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.155182 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.154773 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-credential-socket\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.155182 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.155023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-socket\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.155279 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.155257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istiod-ca-cert\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.156700 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.156678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-envoy\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.157561 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.157541 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-podinfo\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.163814 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.163789 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tfc\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-kube-api-access-f7tfc\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.164121 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.164104 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-token\") pod \"e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.287996 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.287906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:25.430190 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.430167 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4"] Jul 24 17:56:25.431596 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:56:25.431569 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7116bdab_fb07_4018_ac77_ce3a3ff68e89.slice/crio-08ecf0362f3d0e3982ad74a1a736740d428e87143b34fd5a0113fa2e48e23552 WatchSource:0}: Error finding container 08ecf0362f3d0e3982ad74a1a736740d428e87143b34fd5a0113fa2e48e23552: Status 404 returned error can't find the container with id 08ecf0362f3d0e3982ad74a1a736740d428e87143b34fd5a0113fa2e48e23552 Jul 24 17:56:25.433971 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.433927 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:25.434090 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.434023 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:25.434090 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:25.434079 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:26.359961 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:26.359925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" event={"ID":"7116bdab-fb07-4018-ac77-ce3a3ff68e89","Type":"ContainerStarted","Data":"aef95b50f520429adfccf4f08be64f267949a75e604f5f63fc182ec154d48b1c"} Jul 24 17:56:26.360404 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:26.359981 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" event={"ID":"7116bdab-fb07-4018-ac77-ce3a3ff68e89","Type":"ContainerStarted","Data":"08ecf0362f3d0e3982ad74a1a736740d428e87143b34fd5a0113fa2e48e23552"} Jul 24 17:56:26.381130 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:26.381076 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" podStartSLOduration=2.381062506 podStartE2EDuration="2.381062506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:56:26.380851096 +0000 UTC m=+2343.046425735" watchObservedRunningTime="2026-07-24 17:56:26.381062506 +0000 UTC m=+2343.046637142" Jul 24 17:56:27.288218 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:27.288177 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:27.293126 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:27.293104 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:27.365813 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:27.365781 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:27.366758 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:27.366718 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:31.570842 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:31.570807 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:56:33.620413 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.620377 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr"] Jul 24 17:56:33.624314 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.624294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.627657 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.627638 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3922c3-openshift-default-dockercfg-rr7kc\"" Jul 24 17:56:33.660824 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.656714 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr"] Jul 24 17:56:33.753751 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.753212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-certs\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.753751 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.753262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-data\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.753751 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.753320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istiod-ca-cert\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.753751 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.753395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-token\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.753751 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.753461 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-socket\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.753751 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.753488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-credential-socket\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.753751 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.753512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-podinfo\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.753751 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.753537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hqht\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-kube-api-access-6hqht\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.753751 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.753569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-envoy\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.855867 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.855831 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istiod-ca-cert\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.856161 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.856121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istiod-ca-cert\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.856408 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.856391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-token\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.857137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.857116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-socket\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.857489 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.857472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-socket\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.857675 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.857648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-credential-socket\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.858134 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.858115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-podinfo\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.858280 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.858265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hqht\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-kube-api-access-6hqht\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.858418 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.858403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-envoy\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.858578 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.858562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-certs\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.858689 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.858674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-data\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.858860 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.858057 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-credential-socket\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.858959 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.858854 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-certs\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.859052 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.859026 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-data\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.861499 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.861473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-envoy\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.863451 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.863426 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-podinfo\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.867043 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.867023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-token\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.868021 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.867999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hqht\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-kube-api-access-6hqht\") pod \"e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:33.962053 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:33.962010 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:34.129559 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:34.129536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr"] Jul 24 17:56:34.131274 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:56:34.131243 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f44377_c9c5_4ba7_99b7_7dd3e62db529.slice/crio-521fa416a3f56720b14cc8676170362e659f148f432293a963e46f0f65bbf590 WatchSource:0}: Error finding container 521fa416a3f56720b14cc8676170362e659f148f432293a963e46f0f65bbf590: Status 404 returned error can't find the container with id 521fa416a3f56720b14cc8676170362e659f148f432293a963e46f0f65bbf590 Jul 24 17:56:34.134586 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:34.133818 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:34.134586 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:34.133914 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:34.134586 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:34.133958 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:34.407653 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:34.407605 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" event={"ID":"c6f44377-c9c5-4ba7-99b7-7dd3e62db529","Type":"ContainerStarted","Data":"34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f"} Jul 24 17:56:34.407842 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:34.407671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" event={"ID":"c6f44377-c9c5-4ba7-99b7-7dd3e62db529","Type":"ContainerStarted","Data":"521fa416a3f56720b14cc8676170362e659f148f432293a963e46f0f65bbf590"} Jul 24 17:56:34.430923 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:34.430873 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" podStartSLOduration=1.430858022 podStartE2EDuration="1.430858022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:56:34.42835339 +0000 UTC m=+2351.093928029" watchObservedRunningTime="2026-07-24 17:56:34.430858022 +0000 UTC m=+2351.096432659" Jul 24 17:56:34.963156 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:34.963110 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:35.968150 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:35.968121 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:36.421557 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:36.421477 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:36.422669 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:36.422649 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:40.448410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:40.448376 2574 generic.go:356] "Generic (PLEG): container finished" podID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerID="aef95b50f520429adfccf4f08be64f267949a75e604f5f63fc182ec154d48b1c" exitCode=0 Jul 24 17:56:40.448842 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:40.448449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" event={"ID":"7116bdab-fb07-4018-ac77-ce3a3ff68e89","Type":"ContainerDied","Data":"aef95b50f520429adfccf4f08be64f267949a75e604f5f63fc182ec154d48b1c"} Jul 24 17:56:40.448890 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:40.448850 2574 scope.go:117] "RemoveContainer" containerID="aef95b50f520429adfccf4f08be64f267949a75e604f5f63fc182ec154d48b1c" Jul 24 17:56:40.449899 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:40.449867 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:40.450002 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:40.449947 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:40.450002 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:40.449984 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:41.456243 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:41.456209 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" event={"ID":"7116bdab-fb07-4018-ac77-ce3a3ff68e89","Type":"ContainerStarted","Data":"b9e78dd8bdfa908ce935b090e06fe1950196c6fef4f2dbd230734446a8fee834"} Jul 24 17:56:42.288255 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:42.288216 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:42.289565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:42.289540 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 24 17:56:42.289678 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:42.289597 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 24 17:56:42.365847 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:42.365813 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:43.289328 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:43.289293 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 24 17:56:43.289755 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:43.289351 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 24 17:56:43.976764 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:43.976714 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:56:44.288522 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:44.288428 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 24 17:56:44.288522 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:44.288499 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 24 17:56:45.288598 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:45.288549 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 24 17:56:45.289092 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:45.288618 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 24 17:56:46.293281 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:46.293252 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:46.294292 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:46.294274 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:56:52.021611 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:52.021566 2574 patch_prober.go:28] interesting pod/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": read tcp 10.133.0.2:44492->10.133.0.69:15021: read: connection reset by peer" start-of-body= Jul 24 17:56:52.022051 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:52.021677 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": read tcp 10.133.0.2:44492->10.133.0.69:15021: read: connection reset by peer" Jul 24 17:56:52.531302 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:52.531265 2574 generic.go:356] "Generic (PLEG): container finished" podID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerID="34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f" exitCode=0 Jul 24 17:56:52.531493 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:52.531337 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" event={"ID":"c6f44377-c9c5-4ba7-99b7-7dd3e62db529","Type":"ContainerDied","Data":"34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f"} Jul 24 17:56:52.531810 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:52.531793 2574 scope.go:117] "RemoveContainer" containerID="34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f" Jul 24 17:56:52.532797 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:52.532769 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:52.532891 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:52.532858 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:52.532935 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:52.532892 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:56:53.543753 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:53.542966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" event={"ID":"c6f44377-c9c5-4ba7-99b7-7dd3e62db529","Type":"ContainerStarted","Data":"206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f"} Jul 24 17:56:53.963295 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:53.963259 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:53.963547 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:53.963522 2574 patch_prober.go:28] interesting pod/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 24 17:56:53.963633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:53.963614 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 24 17:56:54.963194 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:54.963161 2574 patch_prober.go:28] interesting pod/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 24 17:56:54.963651 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:54.963223 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 24 17:56:55.962647 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:55.962609 2574 patch_prober.go:28] interesting pod/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 24 17:56:55.962869 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:55.962669 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 24 17:56:56.775990 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:56.775953 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:56:56.963365 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:56.963320 2574 patch_prober.go:28] interesting pod/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 24 17:56:56.963571 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:56.963389 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 24 17:56:57.966334 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:57.966301 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:57.966819 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:57.966580 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:56:57.967296 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:56:57.967280 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:57:07.371971 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:57:07.371927 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:57:24.004978 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:57:24.004951 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:57:24.012625 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:57:24.012604 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 17:57:24.389986 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:57:24.389897 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:57:38.171001 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:57:38.170966 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:58:07.353480 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.353434 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz"] Jul 24 17:58:07.357282 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.357256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" Jul 24 17:58:07.361033 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.361011 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:58:07.366446 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.366419 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz"] Jul 24 17:58:07.448637 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.448587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d5fe89-7048-447b-a20b-05888b1f6e90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz\" (UID: \"26d5fe89-7048-447b-a20b-05888b1f6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" Jul 24 17:58:07.549904 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.549865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d5fe89-7048-447b-a20b-05888b1f6e90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz\" (UID: \"26d5fe89-7048-447b-a20b-05888b1f6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" Jul 24 17:58:07.550565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.550544 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d5fe89-7048-447b-a20b-05888b1f6e90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz\" (UID: \"26d5fe89-7048-447b-a20b-05888b1f6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" Jul 24 17:58:07.670053 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.669962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" Jul 24 17:58:07.825683 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.825634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz"] Jul 24 17:58:07.827377 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:58:07.827351 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d5fe89_7048_447b_a20b_05888b1f6e90.slice/crio-d92334e778339c09d7540c9ab012020af156ef809ddde9a3b6cf90c41e9b4ece WatchSource:0}: Error finding container d92334e778339c09d7540c9ab012020af156ef809ddde9a3b6cf90c41e9b4ece: Status 404 returned error can't find the container with id d92334e778339c09d7540c9ab012020af156ef809ddde9a3b6cf90c41e9b4ece Jul 24 17:58:07.986217 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:07.986180 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" event={"ID":"26d5fe89-7048-447b-a20b-05888b1f6e90","Type":"ContainerStarted","Data":"d92334e778339c09d7540c9ab012020af156ef809ddde9a3b6cf90c41e9b4ece"} Jul 24 17:58:08.994228 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:08.994189 2574 generic.go:356] "Generic (PLEG): container finished" podID="26d5fe89-7048-447b-a20b-05888b1f6e90" containerID="c6ed9baed08856524e59198724e857a62ca4e2ed2cbe5e9327e71f5de6929380" exitCode=0 Jul 24 17:58:08.994645 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:08.994272 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" event={"ID":"26d5fe89-7048-447b-a20b-05888b1f6e90","Type":"ContainerDied","Data":"c6ed9baed08856524e59198724e857a62ca4e2ed2cbe5e9327e71f5de6929380"} Jul 24 17:58:10.139558 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:10.139533 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" Jul 24 17:58:10.277027 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:10.276926 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d5fe89-7048-447b-a20b-05888b1f6e90-maas-api-service-ca\") pod \"26d5fe89-7048-447b-a20b-05888b1f6e90\" (UID: \"26d5fe89-7048-447b-a20b-05888b1f6e90\") " Jul 24 17:58:10.277310 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:10.277273 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d5fe89-7048-447b-a20b-05888b1f6e90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26d5fe89-7048-447b-a20b-05888b1f6e90" (UID: "26d5fe89-7048-447b-a20b-05888b1f6e90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:58:10.378319 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:10.378260 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d5fe89-7048-447b-a20b-05888b1f6e90-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:11.007409 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:11.007378 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" event={"ID":"26d5fe89-7048-447b-a20b-05888b1f6e90","Type":"ContainerDied","Data":"d92334e778339c09d7540c9ab012020af156ef809ddde9a3b6cf90c41e9b4ece"} Jul 24 17:58:11.007599 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:11.007432 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92334e778339c09d7540c9ab012020af156ef809ddde9a3b6cf90c41e9b4ece" Jul 24 17:58:11.007599 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:11.007389 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz" Jul 24 17:58:11.190158 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:11.190126 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:58:22.460772 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:22.460738 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz"] Jul 24 17:58:22.463014 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:22.462983 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cdf40f-zw8mz"] Jul 24 17:58:23.803386 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:23.803357 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d5fe89-7048-447b-a20b-05888b1f6e90" path="/var/lib/kubelet/pods/26d5fe89-7048-447b-a20b-05888b1f6e90/volumes" Jul 24 17:58:24.130779 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:24.130655 2574 scope.go:117] "RemoveContainer" containerID="fc92a7e3710a80c2115795f57b40d10f41c2a24db448646c1703cb279268f4c1" Jul 24 17:58:27.187044 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:27.187007 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:58:29.929562 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:29.929525 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4"] Jul 24 17:58:29.930004 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:29.929754 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" containerID="cri-o://b9e78dd8bdfa908ce935b090e06fe1950196c6fef4f2dbd230734446a8fee834" gracePeriod=30 Jul 24 17:58:30.305476 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.305438 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr"] Jul 24 17:58:30.305924 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.305911 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d5fe89-7048-447b-a20b-05888b1f6e90" containerName="revoke-keys" Jul 24 17:58:30.305982 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.305928 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d5fe89-7048-447b-a20b-05888b1f6e90" containerName="revoke-keys" Jul 24 17:58:30.306027 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.306018 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d5fe89-7048-447b-a20b-05888b1f6e90" containerName="revoke-keys" Jul 24 17:58:30.309159 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.309144 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" Jul 24 17:58:30.312377 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.312359 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:58:30.319993 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.319970 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr"] Jul 24 17:58:30.369789 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.369720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1850be2-8826-4f8a-aaf9-111b70f883c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr\" (UID: \"c1850be2-8826-4f8a-aaf9-111b70f883c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" Jul 24 17:58:30.470353 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.470316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1850be2-8826-4f8a-aaf9-111b70f883c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr\" (UID: \"c1850be2-8826-4f8a-aaf9-111b70f883c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" Jul 24 17:58:30.470930 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.470910 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1850be2-8826-4f8a-aaf9-111b70f883c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr\" (UID: \"c1850be2-8826-4f8a-aaf9-111b70f883c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" Jul 24 17:58:30.620766 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.620666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" Jul 24 17:58:30.776109 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:58:30.776079 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1850be2_8826_4f8a_aaf9_111b70f883c1.slice/crio-26669b2ba071e5a7fe691e13f798071f1541937746d8ae9ca991769dd29f3cb2 WatchSource:0}: Error finding container 26669b2ba071e5a7fe691e13f798071f1541937746d8ae9ca991769dd29f3cb2: Status 404 returned error can't find the container with id 26669b2ba071e5a7fe691e13f798071f1541937746d8ae9ca991769dd29f3cb2 Jul 24 17:58:30.776480 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:30.776460 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr"] Jul 24 17:58:31.143897 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:31.143853 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" event={"ID":"c1850be2-8826-4f8a-aaf9-111b70f883c1","Type":"ContainerStarted","Data":"99059317625b1e28c21c86a05ecc610ae38a1729015a869525160fb5c74b0c16"} Jul 24 17:58:31.144272 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:31.143924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" event={"ID":"c1850be2-8826-4f8a-aaf9-111b70f883c1","Type":"ContainerStarted","Data":"26669b2ba071e5a7fe691e13f798071f1541937746d8ae9ca991769dd29f3cb2"} Jul 24 17:58:31.179603 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:31.179550 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" podStartSLOduration=1.1795338069999999 podStartE2EDuration="1.179533807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:58:31.177995238 +0000 UTC m=+2467.843569875" watchObservedRunningTime="2026-07-24 17:58:31.179533807 +0000 UTC m=+2467.845108444" Jul 24 17:58:31.296416 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:31.296382 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 17:58:31.296603 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:31.296441 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:58:32.150456 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:32.150417 2574 generic.go:356] "Generic (PLEG): container finished" podID="c1850be2-8826-4f8a-aaf9-111b70f883c1" containerID="99059317625b1e28c21c86a05ecc610ae38a1729015a869525160fb5c74b0c16" exitCode=0 Jul 24 17:58:32.150889 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:32.150501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" event={"ID":"c1850be2-8826-4f8a-aaf9-111b70f883c1","Type":"ContainerDied","Data":"99059317625b1e28c21c86a05ecc610ae38a1729015a869525160fb5c74b0c16"} Jul 24 17:58:33.289101 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:33.289073 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" Jul 24 17:58:33.398260 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:33.398220 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1850be2-8826-4f8a-aaf9-111b70f883c1-maas-api-service-ca\") pod \"c1850be2-8826-4f8a-aaf9-111b70f883c1\" (UID: \"c1850be2-8826-4f8a-aaf9-111b70f883c1\") " Jul 24 17:58:33.398560 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:33.398539 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1850be2-8826-4f8a-aaf9-111b70f883c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1850be2-8826-4f8a-aaf9-111b70f883c1" (UID: "c1850be2-8826-4f8a-aaf9-111b70f883c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:58:33.499767 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:33.499719 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1850be2-8826-4f8a-aaf9-111b70f883c1-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:34.162423 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:34.162326 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" Jul 24 17:58:34.162423 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:34.162367 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr" event={"ID":"c1850be2-8826-4f8a-aaf9-111b70f883c1","Type":"ContainerDied","Data":"26669b2ba071e5a7fe691e13f798071f1541937746d8ae9ca991769dd29f3cb2"} Jul 24 17:58:34.162635 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:34.162460 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26669b2ba071e5a7fe691e13f798071f1541937746d8ae9ca991769dd29f3cb2" Jul 24 17:58:35.170651 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.170626 2574 generic.go:356] "Generic (PLEG): container finished" podID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerID="b9e78dd8bdfa908ce935b090e06fe1950196c6fef4f2dbd230734446a8fee834" exitCode=0 Jul 24 17:58:35.171026 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.170698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" event={"ID":"7116bdab-fb07-4018-ac77-ce3a3ff68e89","Type":"ContainerDied","Data":"b9e78dd8bdfa908ce935b090e06fe1950196c6fef4f2dbd230734446a8fee834"} Jul 24 17:58:35.171026 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.170789 2574 scope.go:117] "RemoveContainer" containerID="aef95b50f520429adfccf4f08be64f267949a75e604f5f63fc182ec154d48b1c" Jul 24 17:58:35.192416 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.192398 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:58:35.313193 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313157 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-credential-socket\") pod \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " Jul 24 17:58:35.313396 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313203 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-podinfo\") pod \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " Jul 24 17:58:35.313396 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313251 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7tfc\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-kube-api-access-f7tfc\") pod \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " Jul 24 17:58:35.313396 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313306 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-data\") pod \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " Jul 24 17:58:35.313396 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313333 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-socket\") pod \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " Jul 24 17:58:35.313396 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313362 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istiod-ca-cert\") pod \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " Jul 24 17:58:35.313396 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313385 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-certs\") pod \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " Jul 24 17:58:35.313700 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313431 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-token\") pod \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " Jul 24 17:58:35.313700 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313471 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-envoy\") pod \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\" (UID: \"7116bdab-fb07-4018-ac77-ce3a3ff68e89\") " Jul 24 17:58:35.313830 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313465 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7116bdab-fb07-4018-ac77-ce3a3ff68e89" (UID: "7116bdab-fb07-4018-ac77-ce3a3ff68e89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:35.313880 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313714 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7116bdab-fb07-4018-ac77-ce3a3ff68e89" (UID: "7116bdab-fb07-4018-ac77-ce3a3ff68e89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:35.313880 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313860 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7116bdab-fb07-4018-ac77-ce3a3ff68e89" (UID: "7116bdab-fb07-4018-ac77-ce3a3ff68e89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:35.313972 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.313926 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7116bdab-fb07-4018-ac77-ce3a3ff68e89" (UID: "7116bdab-fb07-4018-ac77-ce3a3ff68e89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:58:35.314533 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.314509 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-data" (OuterVolumeSpecName: "istio-data") pod "7116bdab-fb07-4018-ac77-ce3a3ff68e89" (UID: "7116bdab-fb07-4018-ac77-ce3a3ff68e89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:35.315904 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.315874 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7116bdab-fb07-4018-ac77-ce3a3ff68e89" (UID: "7116bdab-fb07-4018-ac77-ce3a3ff68e89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:35.316008 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.315963 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7116bdab-fb07-4018-ac77-ce3a3ff68e89" (UID: "7116bdab-fb07-4018-ac77-ce3a3ff68e89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:58:35.316208 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.316187 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-token" (OuterVolumeSpecName: "istio-token") pod "7116bdab-fb07-4018-ac77-ce3a3ff68e89" (UID: "7116bdab-fb07-4018-ac77-ce3a3ff68e89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:58:35.316255 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.316190 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-kube-api-access-f7tfc" (OuterVolumeSpecName: "kube-api-access-f7tfc") pod "7116bdab-fb07-4018-ac77-ce3a3ff68e89" (UID: "7116bdab-fb07-4018-ac77-ce3a3ff68e89"). InnerVolumeSpecName "kube-api-access-f7tfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:58:35.415065 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.415029 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:35.415065 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.415057 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:35.415065 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.415066 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:35.415065 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.415075 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:35.415336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.415084 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:35.415336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.415091 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:35.415336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.415098 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116bdab-fb07-4018-ac77-ce3a3ff68e89-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:35.415336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.415106 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116bdab-fb07-4018-ac77-ce3a3ff68e89-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:35.415336 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:35.415114 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7tfc\" (UniqueName: \"kubernetes.io/projected/7116bdab-fb07-4018-ac77-ce3a3ff68e89-kube-api-access-f7tfc\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:36.177654 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:36.177573 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" Jul 24 17:58:36.177654 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:36.177585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4" event={"ID":"7116bdab-fb07-4018-ac77-ce3a3ff68e89","Type":"ContainerDied","Data":"08ecf0362f3d0e3982ad74a1a736740d428e87143b34fd5a0113fa2e48e23552"} Jul 24 17:58:36.177654 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:36.177640 2574 scope.go:117] "RemoveContainer" containerID="b9e78dd8bdfa908ce935b090e06fe1950196c6fef4f2dbd230734446a8fee834" Jul 24 17:58:36.217984 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:36.217953 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4"] Jul 24 17:58:36.224401 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:36.224370 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cdf40f-openshift-default-c7c4c47b4-xmmj4"] Jul 24 17:58:37.801451 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:37.801416 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" path="/var/lib/kubelet/pods/7116bdab-fb07-4018-ac77-ce3a3ff68e89/volumes" Jul 24 17:58:43.670076 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:43.670030 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:58:45.389189 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:45.389150 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr"] Jul 24 17:58:45.392192 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:45.392169 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3922c3-gsncr"] Jul 24 17:58:45.801717 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:45.801684 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1850be2-8826-4f8a-aaf9-111b70f883c1" path="/var/lib/kubelet/pods/c1850be2-8826-4f8a-aaf9-111b70f883c1/volumes" Jul 24 17:58:53.256415 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:53.256378 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr"] Jul 24 17:58:53.256921 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:53.256590 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" containerID="cri-o://206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f" gracePeriod=30 Jul 24 17:58:54.526180 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.526146 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6"] Jul 24 17:58:54.526752 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.526711 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" Jul 24 17:58:54.526752 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.526750 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" Jul 24 17:58:54.526905 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.526781 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" Jul 24 17:58:54.526905 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.526790 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" Jul 24 17:58:54.526905 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.526799 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1850be2-8826-4f8a-aaf9-111b70f883c1" containerName="revoke-keys" Jul 24 17:58:54.526905 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.526807 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1850be2-8826-4f8a-aaf9-111b70f883c1" containerName="revoke-keys" Jul 24 17:58:54.527100 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.526918 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" Jul 24 17:58:54.527100 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.526934 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1850be2-8826-4f8a-aaf9-111b70f883c1" containerName="revoke-keys" Jul 24 17:58:54.531827 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.531807 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.548176 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.548152 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5e940d-openshift-default-dockercfg-p8md7\"" Jul 24 17:58:54.575266 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.575236 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6"] Jul 24 17:58:54.686624 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.686592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-credential-socket\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.686624 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.686627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qgj8\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-kube-api-access-7qgj8\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.686877 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.686758 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-envoy\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.686877 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.686792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-socket\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.686877 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.686815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-certs\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.686877 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.686849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ed6843-1198-46b4-96fe-75aef53d39ae-istiod-ca-cert\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.687062 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.686888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-data\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.687062 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.686912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-token\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.687062 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.686960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-podinfo\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.787551 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-podinfo\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.787551 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-credential-socket\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.787551 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qgj8\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-kube-api-access-7qgj8\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.787889 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787664 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-envoy\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.787889 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-socket\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.787889 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787770 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-certs\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.787889 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ed6843-1198-46b4-96fe-75aef53d39ae-istiod-ca-cert\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.787889 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-data\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.788126 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-token\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.788126 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.787946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-credential-socket\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.788126 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.788016 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-socket\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.788284 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.788263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-data\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.788516 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.788497 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-certs\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.788566 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.788501 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ed6843-1198-46b4-96fe-75aef53d39ae-istiod-ca-cert\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.790119 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.790098 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-envoy\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.790342 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.790322 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-podinfo\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.799581 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.799556 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qgj8\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-kube-api-access-7qgj8\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.799705 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.799689 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-token\") pod \"e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:54.842493 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:54.842462 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:55.009683 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:55.009648 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6"] Jul 24 17:58:55.013214 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:58:55.013187 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ed6843_1198_46b4_96fe_75aef53d39ae.slice/crio-291a386ee348688077e4b83b1625ff8ae851c1139562b66c166cdb486a745bcd WatchSource:0}: Error finding container 291a386ee348688077e4b83b1625ff8ae851c1139562b66c166cdb486a745bcd: Status 404 returned error can't find the container with id 291a386ee348688077e4b83b1625ff8ae851c1139562b66c166cdb486a745bcd Jul 24 17:58:55.015330 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:55.015299 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:58:55.015402 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:55.015369 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:58:55.015402 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:55.015398 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:58:55.292201 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:55.292164 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" event={"ID":"e8ed6843-1198-46b4-96fe-75aef53d39ae","Type":"ContainerStarted","Data":"91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b"} Jul 24 17:58:55.292407 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:55.292233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" event={"ID":"e8ed6843-1198-46b4-96fe-75aef53d39ae","Type":"ContainerStarted","Data":"291a386ee348688077e4b83b1625ff8ae851c1139562b66c166cdb486a745bcd"} Jul 24 17:58:55.319949 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:55.319842 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" podStartSLOduration=1.3198267399999999 podStartE2EDuration="1.31982674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:58:55.317572377 +0000 UTC m=+2491.983147017" watchObservedRunningTime="2026-07-24 17:58:55.31982674 +0000 UTC m=+2491.985401377" Jul 24 17:58:55.842908 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:55.842875 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:56.847720 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:56.847686 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:57.303852 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:57.303813 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:57.304865 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:57.304844 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:58:57.969511 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:57.969476 2574 patch_prober.go:28] interesting pod/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 17:58:57.969937 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:57.969539 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:58:58.543671 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.543640 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:58:58.728901 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.728868 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-token\") pod \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " Jul 24 17:58:58.729095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.728910 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-envoy\") pod \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " Jul 24 17:58:58.729095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.728927 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-credential-socket\") pod \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " Jul 24 17:58:58.729095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.728987 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-podinfo\") pod \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " Jul 24 17:58:58.729095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729017 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-certs\") pod \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " Jul 24 17:58:58.729095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729056 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istiod-ca-cert\") pod \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " Jul 24 17:58:58.729095 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729087 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hqht\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-kube-api-access-6hqht\") pod \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " Jul 24 17:58:58.729439 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729122 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-data\") pod \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " Jul 24 17:58:58.729439 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729143 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-socket\") pod \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\" (UID: \"c6f44377-c9c5-4ba7-99b7-7dd3e62db529\") " Jul 24 17:58:58.729439 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729272 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6f44377-c9c5-4ba7-99b7-7dd3e62db529" (UID: "c6f44377-c9c5-4ba7-99b7-7dd3e62db529"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:58.729439 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729283 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6f44377-c9c5-4ba7-99b7-7dd3e62db529" (UID: "c6f44377-c9c5-4ba7-99b7-7dd3e62db529"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:58.729846 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729451 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:58.729846 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729471 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:58.729846 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729474 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6f44377-c9c5-4ba7-99b7-7dd3e62db529" (UID: "c6f44377-c9c5-4ba7-99b7-7dd3e62db529"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:58.729846 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.729544 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6f44377-c9c5-4ba7-99b7-7dd3e62db529" (UID: "c6f44377-c9c5-4ba7-99b7-7dd3e62db529"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:58:58.730291 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.730248 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6f44377-c9c5-4ba7-99b7-7dd3e62db529" (UID: "c6f44377-c9c5-4ba7-99b7-7dd3e62db529"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:58.731486 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.731465 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6f44377-c9c5-4ba7-99b7-7dd3e62db529" (UID: "c6f44377-c9c5-4ba7-99b7-7dd3e62db529"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:58:58.731486 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.731476 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6f44377-c9c5-4ba7-99b7-7dd3e62db529" (UID: "c6f44377-c9c5-4ba7-99b7-7dd3e62db529"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:58:58.731615 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.731578 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-kube-api-access-6hqht" (OuterVolumeSpecName: "kube-api-access-6hqht") pod "c6f44377-c9c5-4ba7-99b7-7dd3e62db529" (UID: "c6f44377-c9c5-4ba7-99b7-7dd3e62db529"). InnerVolumeSpecName "kube-api-access-6hqht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:58:58.731654 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.731603 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6f44377-c9c5-4ba7-99b7-7dd3e62db529" (UID: "c6f44377-c9c5-4ba7-99b7-7dd3e62db529"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:58:58.831064 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.830972 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:58.831064 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.831000 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:58.831064 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.831009 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:58.831064 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.831018 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:58.831064 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.831025 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hqht\" (UniqueName: \"kubernetes.io/projected/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-kube-api-access-6hqht\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:58.831064 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.831035 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:58.831064 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:58.831043 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f44377-c9c5-4ba7-99b7-7dd3e62db529-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:58:59.317082 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.317048 2574 generic.go:356] "Generic (PLEG): container finished" podID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerID="206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f" exitCode=0 Jul 24 17:58:59.317550 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.317095 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" Jul 24 17:58:59.317550 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.317127 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" event={"ID":"c6f44377-c9c5-4ba7-99b7-7dd3e62db529","Type":"ContainerDied","Data":"206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f"} Jul 24 17:58:59.317550 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.317174 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr" event={"ID":"c6f44377-c9c5-4ba7-99b7-7dd3e62db529","Type":"ContainerDied","Data":"521fa416a3f56720b14cc8676170362e659f148f432293a963e46f0f65bbf590"} Jul 24 17:58:59.317550 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.317196 2574 scope.go:117] "RemoveContainer" containerID="206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f" Jul 24 17:58:59.327438 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.327420 2574 scope.go:117] "RemoveContainer" containerID="34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f" Jul 24 17:58:59.336361 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.336342 2574 scope.go:117] "RemoveContainer" containerID="206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f" Jul 24 17:58:59.336627 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:58:59.336607 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f\": container with ID starting with 206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f not found: ID does not exist" containerID="206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f" Jul 24 17:58:59.336703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.336640 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f"} err="failed to get container status \"206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f\": rpc error: code = NotFound desc = could not find container \"206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f\": container with ID starting with 206ba2c2c89d90061c995256689322ab72bfc958ea350db45ff010305c792f3f not found: ID does not exist" Jul 24 17:58:59.336703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.336666 2574 scope.go:117] "RemoveContainer" containerID="34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f" Jul 24 17:58:59.336971 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:58:59.336954 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f\": container with ID starting with 34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f not found: ID does not exist" containerID="34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f" Jul 24 17:58:59.337025 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.336981 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f"} err="failed to get container status \"34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f\": rpc error: code = NotFound desc = could not find container \"34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f\": container with ID starting with 34aa2b4805cd85575576341e50a04c3263a3a0548e100cf482eceb8f098c4f4f not found: ID does not exist" Jul 24 17:58:59.347424 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.347399 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr"] Jul 24 17:58:59.353921 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.353899 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3922c3-openshift-default-9c8c66888-n67fr"] Jul 24 17:58:59.803164 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:58:59.803131 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" path="/var/lib/kubelet/pods/c6f44377-c9c5-4ba7-99b7-7dd3e62db529/volumes" Jul 24 17:59:02.916257 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.916215 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b"] Jul 24 17:59:02.916643 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.916602 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" Jul 24 17:59:02.916643 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.916613 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" Jul 24 17:59:02.916643 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.916631 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" Jul 24 17:59:02.916643 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.916636 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" Jul 24 17:59:02.916826 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.916703 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" Jul 24 17:59:02.916826 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.916710 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7116bdab-fb07-4018-ac77-ce3a3ff68e89" containerName="istio-proxy" Jul 24 17:59:02.916826 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.916719 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f44377-c9c5-4ba7-99b7-7dd3e62db529" containerName="istio-proxy" Jul 24 17:59:02.921484 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.921464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:02.925633 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.925612 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0ed7b2-openshift-default-dockercfg-z8r9h\"" Jul 24 17:59:02.963907 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.963874 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b"] Jul 24 17:59:02.966587 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.966562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1abe95-9939-4d31-b756-321a446bc9fa-istiod-ca-cert\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:02.966688 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.966597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-envoy\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:02.966688 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.966616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-data\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:02.966688 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.966631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-podinfo\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:02.966836 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.966760 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-socket\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:02.966836 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.966824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-certs\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:02.966913 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.966862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-credential-socket\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:02.966913 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.966889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-token\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:02.966989 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:02.966955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvmj\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-kube-api-access-blvmj\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.067915 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.067880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvmj\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-kube-api-access-blvmj\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.067959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1abe95-9939-4d31-b756-321a446bc9fa-istiod-ca-cert\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.067980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-envoy\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.067998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-data\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.068014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-podinfo\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.068039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-socket\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.068073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-certs\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.068106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-credential-socket\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068550 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.068129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-token\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068550 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.068502 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-certs\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068670 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.068646 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-credential-socket\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068759 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.068678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-socket\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.068759 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.068701 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-data\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.069044 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.069006 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1abe95-9939-4d31-b756-321a446bc9fa-istiod-ca-cert\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.070555 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.070532 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-envoy\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.070795 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.070774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-podinfo\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.083071 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.083045 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-token\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.083214 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.083196 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvmj\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-kube-api-access-blvmj\") pod \"e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.232248 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.232207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:03.393841 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.393806 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b"] Jul 24 17:59:03.399045 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:59:03.399008 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1abe95_9939_4d31_b756_321a446bc9fa.slice/crio-cf226cf291a7b0b98af2f32b103a6c1626b1837b0e9cc7c6e332972ad92fe2f9 WatchSource:0}: Error finding container cf226cf291a7b0b98af2f32b103a6c1626b1837b0e9cc7c6e332972ad92fe2f9: Status 404 returned error can't find the container with id cf226cf291a7b0b98af2f32b103a6c1626b1837b0e9cc7c6e332972ad92fe2f9 Jul 24 17:59:03.401319 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.401290 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:59:03.401381 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.401360 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:59:03.401424 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:03.401390 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:59:04.351218 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:04.351183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" event={"ID":"8d1abe95-9939-4d31-b756-321a446bc9fa","Type":"ContainerStarted","Data":"39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6"} Jul 24 17:59:04.351610 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:04.351238 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" event={"ID":"8d1abe95-9939-4d31-b756-321a446bc9fa","Type":"ContainerStarted","Data":"cf226cf291a7b0b98af2f32b103a6c1626b1837b0e9cc7c6e332972ad92fe2f9"} Jul 24 17:59:04.382382 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:04.382325 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" podStartSLOduration=2.382309403 podStartE2EDuration="2.382309403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:59:04.380582548 +0000 UTC m=+2501.046157186" watchObservedRunningTime="2026-07-24 17:59:04.382309403 +0000 UTC m=+2501.047884039" Jul 24 17:59:04.486393 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:04.486351 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:59:05.233323 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:05.233281 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:05.238236 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:05.238207 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:05.362539 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:05.362435 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:05.365397 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:05.365349 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:12.431636 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:12.431509 2574 patch_prober.go:28] interesting pod/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:53390->10.133.0.72:15021: read: connection reset by peer" start-of-body= Jul 24 17:59:12.431636 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:12.431597 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:53390->10.133.0.72:15021: read: connection reset by peer" Jul 24 17:59:13.410872 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:13.410832 2574 generic.go:356] "Generic (PLEG): container finished" podID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerID="91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b" exitCode=0 Jul 24 17:59:13.411053 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:13.410874 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" event={"ID":"e8ed6843-1198-46b4-96fe-75aef53d39ae","Type":"ContainerDied","Data":"91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b"} Jul 24 17:59:13.411294 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:13.411281 2574 scope.go:117] "RemoveContainer" containerID="91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b" Jul 24 17:59:13.412360 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:13.412331 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:59:13.412435 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:13.412421 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:59:13.412473 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:13.412460 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:59:14.418985 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:14.418894 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" event={"ID":"e8ed6843-1198-46b4-96fe-75aef53d39ae","Type":"ContainerStarted","Data":"f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b"} Jul 24 17:59:14.847766 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:14.846130 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:59:14.851748 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:14.848843 2574 patch_prober.go:28] interesting pod/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 24 17:59:14.851748 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:14.848907 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 24 17:59:15.843969 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:15.843925 2574 patch_prober.go:28] interesting pod/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 24 17:59:15.844508 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:15.843994 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 24 17:59:16.705151 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:16.705115 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:59:16.843565 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:16.843527 2574 patch_prober.go:28] interesting pod/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 24 17:59:16.843890 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:16.843857 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 24 17:59:17.843948 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:17.843908 2574 patch_prober.go:28] interesting pod/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 24 17:59:17.845023 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:17.844543 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 24 17:59:18.843696 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:18.843237 2574 patch_prober.go:28] interesting pod/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 24 17:59:18.843696 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:18.843304 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 24 17:59:19.153143 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.153033 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4"] Jul 24 17:59:19.157333 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.157311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" Jul 24 17:59:19.160768 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.160714 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:59:19.164773 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.164747 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4"] Jul 24 17:59:19.221394 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.221353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a0af79-2ea1-435e-9eda-209f07f69c28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4\" (UID: \"a0a0af79-2ea1-435e-9eda-209f07f69c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" Jul 24 17:59:19.322343 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.322306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a0af79-2ea1-435e-9eda-209f07f69c28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4\" (UID: \"a0a0af79-2ea1-435e-9eda-209f07f69c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" Jul 24 17:59:19.323031 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.323006 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a0af79-2ea1-435e-9eda-209f07f69c28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4\" (UID: \"a0a0af79-2ea1-435e-9eda-209f07f69c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" Jul 24 17:59:19.472976 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.472939 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" Jul 24 17:59:19.622612 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.622580 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4"] Jul 24 17:59:19.623563 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:59:19.623538 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a0af79_2ea1_435e_9eda_209f07f69c28.slice/crio-5bcc8491116ffd96f0291dd4aff63625378c6ddaa0f81817b2f4e0b4b24ccb00 WatchSource:0}: Error finding container 5bcc8491116ffd96f0291dd4aff63625378c6ddaa0f81817b2f4e0b4b24ccb00: Status 404 returned error can't find the container with id 5bcc8491116ffd96f0291dd4aff63625378c6ddaa0f81817b2f4e0b4b24ccb00 Jul 24 17:59:19.846401 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.846328 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:59:19.846624 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.846606 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:59:19.847520 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:19.847504 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:59:20.462868 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:20.462837 2574 generic.go:356] "Generic (PLEG): container finished" podID="a0a0af79-2ea1-435e-9eda-209f07f69c28" containerID="af5edb7ca5ab8987feb9ae95b9598b601cbc0893c31a9e84e9d81df3eb423d92" exitCode=0 Jul 24 17:59:20.463326 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:20.462902 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" event={"ID":"a0a0af79-2ea1-435e-9eda-209f07f69c28","Type":"ContainerDied","Data":"af5edb7ca5ab8987feb9ae95b9598b601cbc0893c31a9e84e9d81df3eb423d92"} Jul 24 17:59:20.463326 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:20.462952 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" event={"ID":"a0a0af79-2ea1-435e-9eda-209f07f69c28","Type":"ContainerStarted","Data":"5bcc8491116ffd96f0291dd4aff63625378c6ddaa0f81817b2f4e0b4b24ccb00"} Jul 24 17:59:20.939217 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:20.939187 2574 patch_prober.go:28] interesting pod/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": read tcp 10.133.0.2:59858->10.133.0.73:15021: read: connection reset by peer" start-of-body= Jul 24 17:59:20.939406 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:20.939249 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": read tcp 10.133.0.2:59858->10.133.0.73:15021: read: connection reset by peer" Jul 24 17:59:21.470544 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.470507 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerID="39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6" exitCode=0 Jul 24 17:59:21.470979 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.470578 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" event={"ID":"8d1abe95-9939-4d31-b756-321a446bc9fa","Type":"ContainerDied","Data":"39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6"} Jul 24 17:59:21.471385 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.471218 2574 scope.go:117] "RemoveContainer" containerID="39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6" Jul 24 17:59:21.472250 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.472221 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:59:21.472340 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.472293 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:59:21.472340 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.472333 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 17:59:21.657247 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.657222 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" Jul 24 17:59:21.746196 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.746112 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a0af79-2ea1-435e-9eda-209f07f69c28-maas-api-service-ca\") pod \"a0a0af79-2ea1-435e-9eda-209f07f69c28\" (UID: \"a0a0af79-2ea1-435e-9eda-209f07f69c28\") " Jul 24 17:59:21.746535 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.746508 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a0af79-2ea1-435e-9eda-209f07f69c28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0a0af79-2ea1-435e-9eda-209f07f69c28" (UID: "a0a0af79-2ea1-435e-9eda-209f07f69c28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:59:21.847830 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:21.847794 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a0af79-2ea1-435e-9eda-209f07f69c28-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:22.477877 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:22.477840 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" event={"ID":"8d1abe95-9939-4d31-b756-321a446bc9fa","Type":"ContainerStarted","Data":"a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60"} Jul 24 17:59:22.479260 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:22.479233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" event={"ID":"a0a0af79-2ea1-435e-9eda-209f07f69c28","Type":"ContainerDied","Data":"5bcc8491116ffd96f0291dd4aff63625378c6ddaa0f81817b2f4e0b4b24ccb00"} Jul 24 17:59:22.479362 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:22.479276 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bcc8491116ffd96f0291dd4aff63625378c6ddaa0f81817b2f4e0b4b24ccb00" Jul 24 17:59:22.479362 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:22.479247 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4" Jul 24 17:59:23.233686 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:23.233603 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:23.233686 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:23.233652 2574 patch_prober.go:28] interesting pod/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 24 17:59:23.234000 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:23.233708 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 24 17:59:24.233308 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:24.233260 2574 patch_prober.go:28] interesting pod/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 24 17:59:24.233840 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:24.233327 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 24 17:59:25.233170 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:25.233130 2574 patch_prober.go:28] interesting pod/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 24 17:59:25.233365 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:25.233197 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 24 17:59:26.232577 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:26.232540 2574 patch_prober.go:28] interesting pod/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 24 17:59:26.232842 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:26.232607 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 24 17:59:27.236770 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:27.236716 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:27.237156 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:27.237056 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:27.237707 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:27.237686 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 17:59:37.071063 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:37.071023 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 17:59:39.232408 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:39.232371 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4"] Jul 24 17:59:39.234929 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:39.234901 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5e940d-dc5f4"] Jul 24 17:59:39.807583 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:39.807545 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a0af79-2ea1-435e-9eda-209f07f69c28" path="/var/lib/kubelet/pods/a0a0af79-2ea1-435e-9eda-209f07f69c28/volumes" Jul 24 17:59:46.265022 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.264976 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6"] Jul 24 17:59:46.265443 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.265241 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" containerID="cri-o://f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b" gracePeriod=30 Jul 24 17:59:46.660596 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.660497 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2"] Jul 24 17:59:46.661063 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.661045 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a0af79-2ea1-435e-9eda-209f07f69c28" containerName="revoke-keys" Jul 24 17:59:46.661179 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.661065 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a0af79-2ea1-435e-9eda-209f07f69c28" containerName="revoke-keys" Jul 24 17:59:46.661179 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.661164 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a0af79-2ea1-435e-9eda-209f07f69c28" containerName="revoke-keys" Jul 24 17:59:46.665035 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.665013 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" Jul 24 17:59:46.668096 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.668073 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 17:59:46.672686 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.672664 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2"] Jul 24 17:59:46.690087 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.690046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bd731b-6c67-40f3-9fe3-003a85e75df9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2\" (UID: \"b0bd731b-6c67-40f3-9fe3-003a85e75df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" Jul 24 17:59:46.791547 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.791508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bd731b-6c67-40f3-9fe3-003a85e75df9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2\" (UID: \"b0bd731b-6c67-40f3-9fe3-003a85e75df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" Jul 24 17:59:46.792193 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.792172 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bd731b-6c67-40f3-9fe3-003a85e75df9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2\" (UID: \"b0bd731b-6c67-40f3-9fe3-003a85e75df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" Jul 24 17:59:46.978352 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:46.978317 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" Jul 24 17:59:47.120144 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:47.120118 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2"] Jul 24 17:59:47.122652 ip-10-0-130-58 kubenswrapper[2574]: W0724 17:59:47.122627 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bd731b_6c67_40f3_9fe3_003a85e75df9.slice/crio-f8c4dfb651172b593a109445bff65f0eed3a5f227f3b17dff14894fbd57d3c35 WatchSource:0}: Error finding container f8c4dfb651172b593a109445bff65f0eed3a5f227f3b17dff14894fbd57d3c35: Status 404 returned error can't find the container with id f8c4dfb651172b593a109445bff65f0eed3a5f227f3b17dff14894fbd57d3c35 Jul 24 17:59:47.636801 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:47.636696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" event={"ID":"b0bd731b-6c67-40f3-9fe3-003a85e75df9","Type":"ContainerStarted","Data":"6c4fd61d17b26d3286db4d3907eec1eac54fd8e15bada267e0c42a25d9743cb0"} Jul 24 17:59:47.636801 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:47.636787 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" event={"ID":"b0bd731b-6c67-40f3-9fe3-003a85e75df9","Type":"ContainerStarted","Data":"f8c4dfb651172b593a109445bff65f0eed3a5f227f3b17dff14894fbd57d3c35"} Jul 24 17:59:47.654481 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:47.654423 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" podStartSLOduration=1.654380918 podStartE2EDuration="1.654380918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:59:47.653154473 +0000 UTC m=+2544.318729110" watchObservedRunningTime="2026-07-24 17:59:47.654380918 +0000 UTC m=+2544.319955554" Jul 24 17:59:48.644786 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:48.644749 2574 generic.go:356] "Generic (PLEG): container finished" podID="b0bd731b-6c67-40f3-9fe3-003a85e75df9" containerID="6c4fd61d17b26d3286db4d3907eec1eac54fd8e15bada267e0c42a25d9743cb0" exitCode=0 Jul 24 17:59:48.645225 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:48.644812 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" event={"ID":"b0bd731b-6c67-40f3-9fe3-003a85e75df9","Type":"ContainerDied","Data":"6c4fd61d17b26d3286db4d3907eec1eac54fd8e15bada267e0c42a25d9743cb0"} Jul 24 17:59:49.790432 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:49.790400 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" Jul 24 17:59:49.821498 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:49.821471 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bd731b-6c67-40f3-9fe3-003a85e75df9-maas-api-service-ca\") pod \"b0bd731b-6c67-40f3-9fe3-003a85e75df9\" (UID: \"b0bd731b-6c67-40f3-9fe3-003a85e75df9\") " Jul 24 17:59:49.821838 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:49.821813 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bd731b-6c67-40f3-9fe3-003a85e75df9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0bd731b-6c67-40f3-9fe3-003a85e75df9" (UID: "b0bd731b-6c67-40f3-9fe3-003a85e75df9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:59:49.848529 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:49.848498 2574 patch_prober.go:28] interesting pod/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 17:59:49.848702 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:49.848557 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:59:49.922551 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:49.922463 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bd731b-6c67-40f3-9fe3-003a85e75df9-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:50.658871 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:50.658844 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" Jul 24 17:59:50.659052 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:50.658837 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2" event={"ID":"b0bd731b-6c67-40f3-9fe3-003a85e75df9","Type":"ContainerDied","Data":"f8c4dfb651172b593a109445bff65f0eed3a5f227f3b17dff14894fbd57d3c35"} Jul 24 17:59:50.659052 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:50.658968 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8c4dfb651172b593a109445bff65f0eed3a5f227f3b17dff14894fbd57d3c35" Jul 24 17:59:51.520680 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.520654 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:59:51.640135 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640042 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-token\") pod \"e8ed6843-1198-46b4-96fe-75aef53d39ae\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " Jul 24 17:59:51.640135 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640101 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ed6843-1198-46b4-96fe-75aef53d39ae-istiod-ca-cert\") pod \"e8ed6843-1198-46b4-96fe-75aef53d39ae\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " Jul 24 17:59:51.640376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640137 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qgj8\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-kube-api-access-7qgj8\") pod \"e8ed6843-1198-46b4-96fe-75aef53d39ae\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " Jul 24 17:59:51.640376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640196 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-certs\") pod \"e8ed6843-1198-46b4-96fe-75aef53d39ae\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " Jul 24 17:59:51.640376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640247 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-credential-socket\") pod \"e8ed6843-1198-46b4-96fe-75aef53d39ae\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " Jul 24 17:59:51.640376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640273 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-data\") pod \"e8ed6843-1198-46b4-96fe-75aef53d39ae\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " Jul 24 17:59:51.640376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640305 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-socket\") pod \"e8ed6843-1198-46b4-96fe-75aef53d39ae\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " Jul 24 17:59:51.640376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640333 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-envoy\") pod \"e8ed6843-1198-46b4-96fe-75aef53d39ae\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " Jul 24 17:59:51.640376 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640359 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-podinfo\") pod \"e8ed6843-1198-46b4-96fe-75aef53d39ae\" (UID: \"e8ed6843-1198-46b4-96fe-75aef53d39ae\") " Jul 24 17:59:51.640703 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640579 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ed6843-1198-46b4-96fe-75aef53d39ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8ed6843-1198-46b4-96fe-75aef53d39ae" (UID: "e8ed6843-1198-46b4-96fe-75aef53d39ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:59:51.640977 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640892 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8ed6843-1198-46b4-96fe-75aef53d39ae" (UID: "e8ed6843-1198-46b4-96fe-75aef53d39ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:59:51.641438 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.641380 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ed6843-1198-46b4-96fe-75aef53d39ae-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:51.641438 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.640793 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8ed6843-1198-46b4-96fe-75aef53d39ae" (UID: "e8ed6843-1198-46b4-96fe-75aef53d39ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:59:51.641809 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.641600 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8ed6843-1198-46b4-96fe-75aef53d39ae" (UID: "e8ed6843-1198-46b4-96fe-75aef53d39ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:59:51.641809 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.641679 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8ed6843-1198-46b4-96fe-75aef53d39ae" (UID: "e8ed6843-1198-46b4-96fe-75aef53d39ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:59:51.642663 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.642641 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8ed6843-1198-46b4-96fe-75aef53d39ae" (UID: "e8ed6843-1198-46b4-96fe-75aef53d39ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:59:51.642663 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.642656 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-kube-api-access-7qgj8" (OuterVolumeSpecName: "kube-api-access-7qgj8") pod "e8ed6843-1198-46b4-96fe-75aef53d39ae" (UID: "e8ed6843-1198-46b4-96fe-75aef53d39ae"). InnerVolumeSpecName "kube-api-access-7qgj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:59:51.642930 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.642912 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8ed6843-1198-46b4-96fe-75aef53d39ae" (UID: "e8ed6843-1198-46b4-96fe-75aef53d39ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:59:51.643101 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.643085 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8ed6843-1198-46b4-96fe-75aef53d39ae" (UID: "e8ed6843-1198-46b4-96fe-75aef53d39ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:59:51.666124 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.666098 2574 generic.go:356] "Generic (PLEG): container finished" podID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerID="f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b" exitCode=0 Jul 24 17:59:51.666268 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.666147 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" Jul 24 17:59:51.666268 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.666170 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" event={"ID":"e8ed6843-1198-46b4-96fe-75aef53d39ae","Type":"ContainerDied","Data":"f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b"} Jul 24 17:59:51.666268 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.666214 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6" event={"ID":"e8ed6843-1198-46b4-96fe-75aef53d39ae","Type":"ContainerDied","Data":"291a386ee348688077e4b83b1625ff8ae851c1139562b66c166cdb486a745bcd"} Jul 24 17:59:51.666268 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.666237 2574 scope.go:117] "RemoveContainer" containerID="f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b" Jul 24 17:59:51.676283 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.676267 2574 scope.go:117] "RemoveContainer" containerID="91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b" Jul 24 17:59:51.685804 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.685785 2574 scope.go:117] "RemoveContainer" containerID="f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b" Jul 24 17:59:51.686097 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:59:51.686074 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b\": container with ID starting with f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b not found: ID does not exist" containerID="f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b" Jul 24 17:59:51.686189 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.686105 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b"} err="failed to get container status \"f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b\": rpc error: code = NotFound desc = could not find container \"f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b\": container with ID starting with f4cc9eb7049008df350a94400d90ad69d2ceb2c999243f78eb1d7a64de36c57b not found: ID does not exist" Jul 24 17:59:51.686189 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.686122 2574 scope.go:117] "RemoveContainer" containerID="91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b" Jul 24 17:59:51.686369 ip-10-0-130-58 kubenswrapper[2574]: E0724 17:59:51.686351 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b\": container with ID starting with 91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b not found: ID does not exist" containerID="91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b" Jul 24 17:59:51.686410 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.686371 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b"} err="failed to get container status \"91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b\": rpc error: code = NotFound desc = could not find container \"91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b\": container with ID starting with 91f656b206a546d57d5cd0a72ec5c5c145095beb73b3425a28c4062b2edfe34b not found: ID does not exist" Jul 24 17:59:51.690606 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.690584 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6"] Jul 24 17:59:51.694757 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.694714 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5e940d-openshift-default-664f8ff4d6-vt5v6"] Jul 24 17:59:51.742902 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.742873 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:51.742902 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.742899 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:51.742902 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.742909 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:51.743137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.742919 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:51.743137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.742927 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:51.743137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.742935 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:51.743137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.742942 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:51.743137 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.742950 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qgj8\" (UniqueName: \"kubernetes.io/projected/e8ed6843-1198-46b4-96fe-75aef53d39ae-kube-api-access-7qgj8\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 17:59:51.808701 ip-10-0-130-58 kubenswrapper[2574]: I0724 17:59:51.808667 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" path="/var/lib/kubelet/pods/e8ed6843-1198-46b4-96fe-75aef53d39ae/volumes" Jul 24 18:00:00.140907 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.140867 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2"] Jul 24 18:00:00.141314 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.141299 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0bd731b-6c67-40f3-9fe3-003a85e75df9" containerName="revoke-keys" Jul 24 18:00:00.141363 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.141316 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0bd731b-6c67-40f3-9fe3-003a85e75df9" containerName="revoke-keys" Jul 24 18:00:00.141363 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.141333 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" Jul 24 18:00:00.141363 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.141338 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" Jul 24 18:00:00.141363 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.141345 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" Jul 24 18:00:00.141363 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.141351 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" Jul 24 18:00:00.141513 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.141408 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" Jul 24 18:00:00.141513 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.141415 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0bd731b-6c67-40f3-9fe3-003a85e75df9" containerName="revoke-keys" Jul 24 18:00:00.141513 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.141425 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ed6843-1198-46b4-96fe-75aef53d39ae" containerName="istio-proxy" Jul 24 18:00:00.144376 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.144357 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q"] Jul 24 18:00:00.144492 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.144471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" Jul 24 18:00:00.147898 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.147873 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 18:00:00.148037 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.147992 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q" Jul 24 18:00:00.152584 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.152080 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q"] Jul 24 18:00:00.155802 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.155776 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2"] Jul 24 18:00:00.159335 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.159314 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" Jul 24 18:00:00.167309 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.167288 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q" Jul 24 18:00:00.321185 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.321146 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2"] Jul 24 18:00:00.321966 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:00:00.321939 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e46929_d0ea_46a2_9e73_1ff85c88273f.slice/crio-3c86fc8de5e59bc64eef765a7115c60fd416438679acb2d1b1d0aec6b04718ad WatchSource:0}: Error finding container 3c86fc8de5e59bc64eef765a7115c60fd416438679acb2d1b1d0aec6b04718ad: Status 404 returned error can't find the container with id 3c86fc8de5e59bc64eef765a7115c60fd416438679acb2d1b1d0aec6b04718ad Jul 24 18:00:00.559543 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.559516 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q"] Jul 24 18:00:00.561529 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:00:00.561500 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d673fe5_02f8_4a18_8628_db7d478e221d.slice/crio-728e5e07ef2c446cd5651c667977bb7ef876212b82426bbc587b40c9db0905e2 WatchSource:0}: Error finding container 728e5e07ef2c446cd5651c667977bb7ef876212b82426bbc587b40c9db0905e2: Status 404 returned error can't find the container with id 728e5e07ef2c446cd5651c667977bb7ef876212b82426bbc587b40c9db0905e2 Jul 24 18:00:00.725516 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.725486 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q" event={"ID":"0d673fe5-02f8-4a18-8628-db7d478e221d","Type":"ContainerStarted","Data":"728e5e07ef2c446cd5651c667977bb7ef876212b82426bbc587b40c9db0905e2"} Jul 24 18:00:00.727006 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.726983 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" event={"ID":"53e46929-d0ea-46a2-9e73-1ff85c88273f","Type":"ContainerStarted","Data":"a6f872d0f6ba639c806cc94da7afaf660fbb46fa9d630b95f0913a89cc1235f9"} Jul 24 18:00:00.727138 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.727022 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" event={"ID":"53e46929-d0ea-46a2-9e73-1ff85c88273f","Type":"ContainerStarted","Data":"3c86fc8de5e59bc64eef765a7115c60fd416438679acb2d1b1d0aec6b04718ad"} Jul 24 18:00:00.746031 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:00.745982 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" podStartSLOduration=0.745961739 podStartE2EDuration="745.961739ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:00:00.745821889 +0000 UTC m=+2557.411396526" watchObservedRunningTime="2026-07-24 18:00:00.745961739 +0000 UTC m=+2557.411536377" Jul 24 18:00:01.734665 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:01.734629 2574 generic.go:356] "Generic (PLEG): container finished" podID="0d673fe5-02f8-4a18-8628-db7d478e221d" containerID="278741c60654915448eb77a70aec3e55f765f3c05db9ed2134720f30d0f42606" exitCode=0 Jul 24 18:00:01.735122 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:01.734754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q" event={"ID":"0d673fe5-02f8-4a18-8628-db7d478e221d","Type":"ContainerDied","Data":"278741c60654915448eb77a70aec3e55f765f3c05db9ed2134720f30d0f42606"} Jul 24 18:00:01.736247 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:01.736225 2574 generic.go:356] "Generic (PLEG): container finished" podID="53e46929-d0ea-46a2-9e73-1ff85c88273f" containerID="a6f872d0f6ba639c806cc94da7afaf660fbb46fa9d630b95f0913a89cc1235f9" exitCode=0 Jul 24 18:00:01.736372 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:01.736257 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" event={"ID":"53e46929-d0ea-46a2-9e73-1ff85c88273f","Type":"ContainerDied","Data":"a6f872d0f6ba639c806cc94da7afaf660fbb46fa9d630b95f0913a89cc1235f9"} Jul 24 18:00:02.923308 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:02.923197 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" Jul 24 18:00:02.949620 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:02.949596 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q" Jul 24 18:00:02.967499 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:02.967467 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2"] Jul 24 18:00:02.979370 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:02.979334 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2"] Jul 24 18:00:03.750806 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:03.750766 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q" event={"ID":"0d673fe5-02f8-4a18-8628-db7d478e221d","Type":"ContainerDied","Data":"728e5e07ef2c446cd5651c667977bb7ef876212b82426bbc587b40c9db0905e2"} Jul 24 18:00:03.750806 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:03.750795 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748600-r5g2q" Jul 24 18:00:03.751086 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:03.750829 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="728e5e07ef2c446cd5651c667977bb7ef876212b82426bbc587b40c9db0905e2" Jul 24 18:00:03.752114 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:03.752092 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" Jul 24 18:00:03.752237 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:03.752128 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c86fc8de5e59bc64eef765a7115c60fd416438679acb2d1b1d0aec6b04718ad" Jul 24 18:00:03.766294 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:00:03.766266 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="53e46929-d0ea-46a2-9e73-1ff85c88273f" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" Jul 24 18:00:03.801709 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:03.801679 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53e46929-d0ea-46a2-9e73-1ff85c88273f" path="/var/lib/kubelet/pods/53e46929-d0ea-46a2-9e73-1ff85c88273f/volumes" Jul 24 18:00:03.802670 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:00:03.802645 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="53e46929-d0ea-46a2-9e73-1ff85c88273f" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0ed7b2-29748600-blsm2" Jul 24 18:00:06.748508 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:06.748473 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2"] Jul 24 18:00:06.751573 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:06.751548 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ed7b2-ttsl2"] Jul 24 18:00:07.801648 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:07.801612 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0bd731b-6c67-40f3-9fe3-003a85e75df9" path="/var/lib/kubelet/pods/b0bd731b-6c67-40f3-9fe3-003a85e75df9/volumes" Jul 24 18:00:08.077853 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:08.077777 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:00:13.262464 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:13.262422 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b"] Jul 24 18:00:13.263474 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:13.263374 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" containerID="cri-o://a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60" gracePeriod=30 Jul 24 18:00:14.246156 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.246123 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz"] Jul 24 18:00:14.246538 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.246521 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d673fe5-02f8-4a18-8628-db7d478e221d" containerName="cleanup" Jul 24 18:00:14.246538 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.246536 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d673fe5-02f8-4a18-8628-db7d478e221d" containerName="cleanup" Jul 24 18:00:14.246678 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.246570 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53e46929-d0ea-46a2-9e73-1ff85c88273f" containerName="cleanup" Jul 24 18:00:14.246678 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.246577 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="53e46929-d0ea-46a2-9e73-1ff85c88273f" containerName="cleanup" Jul 24 18:00:14.246678 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.246650 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="53e46929-d0ea-46a2-9e73-1ff85c88273f" containerName="cleanup" Jul 24 18:00:14.246678 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.246658 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d673fe5-02f8-4a18-8628-db7d478e221d" containerName="cleanup" Jul 24 18:00:14.250824 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.250804 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.254019 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.253990 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-257940-openshift-default-dockercfg-zlkxn\"" Jul 24 18:00:14.265087 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.265054 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz"] Jul 24 18:00:14.354481 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.354443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-certs\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.354481 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.354482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-podinfo\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.354760 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.354501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-socket\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.354760 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.354556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rnxh\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-kube-api-access-5rnxh\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.354760 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.354599 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istiod-ca-cert\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.354760 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.354650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-credential-socket\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.354760 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.354666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-data\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.354760 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.354681 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-token\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.354991 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.354788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-envoy\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455359 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-socket\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455359 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rnxh\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-kube-api-access-5rnxh\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455647 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istiod-ca-cert\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455647 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455506 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-credential-socket\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455647 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-data\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455647 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-token\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455647 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-envoy\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455957 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-certs\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455957 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455813 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-socket\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455957 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-podinfo\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.455957 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455917 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-data\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.456124 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.455992 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-credential-socket\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.456124 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.456078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-certs\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.456263 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.456240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istiod-ca-cert\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.458252 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.458226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-envoy\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.458376 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.458264 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-podinfo\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.464408 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.464389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-token\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.464893 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.464874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rnxh\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-kube-api-access-5rnxh\") pod \"e2e-shared-a-257940-openshift-default-845d57f676-rhnlz\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.561246 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.561151 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:14.716362 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.716335 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz"] Jul 24 18:00:14.717578 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:00:14.717542 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6603c3a2_5079_4bf6_bf0a_2f0863a9b51e.slice/crio-fb02418bd8f0c6244d8d9d57c4ddb9f2e20a009e0f1f7059270ad86a66eb9b8b WatchSource:0}: Error finding container fb02418bd8f0c6244d8d9d57c4ddb9f2e20a009e0f1f7059270ad86a66eb9b8b: Status 404 returned error can't find the container with id fb02418bd8f0c6244d8d9d57c4ddb9f2e20a009e0f1f7059270ad86a66eb9b8b Jul 24 18:00:14.719621 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.719589 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:14.719744 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.719666 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:14.719744 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.719714 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:14.820012 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.819921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" event={"ID":"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e","Type":"ContainerStarted","Data":"dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf"} Jul 24 18:00:14.820012 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.819992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" event={"ID":"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e","Type":"ContainerStarted","Data":"fb02418bd8f0c6244d8d9d57c4ddb9f2e20a009e0f1f7059270ad86a66eb9b8b"} Jul 24 18:00:14.841885 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:14.841829 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" podStartSLOduration=0.841807657 podStartE2EDuration="841.807657ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:00:14.841537334 +0000 UTC m=+2571.507111971" watchObservedRunningTime="2026-07-24 18:00:14.841807657 +0000 UTC m=+2571.507382295" Jul 24 18:00:15.561757 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:15.561695 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:15.563163 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:15.563134 2574 patch_prober.go:28] interesting pod/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Jul 24 18:00:15.563287 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:15.563184 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Jul 24 18:00:16.565712 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:16.565676 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:16.832358 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:16.832275 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:16.833167 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:16.833143 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:18.536810 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.536715 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 18:00:18.698632 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.698600 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1abe95-9939-4d31-b756-321a446bc9fa-istiod-ca-cert\") pod \"8d1abe95-9939-4d31-b756-321a446bc9fa\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " Jul 24 18:00:18.698839 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.698646 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-socket\") pod \"8d1abe95-9939-4d31-b756-321a446bc9fa\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " Jul 24 18:00:18.698839 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.698672 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-token\") pod \"8d1abe95-9939-4d31-b756-321a446bc9fa\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " Jul 24 18:00:18.698839 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.698765 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-podinfo\") pod \"8d1abe95-9939-4d31-b756-321a446bc9fa\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " Jul 24 18:00:18.698839 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.698828 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blvmj\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-kube-api-access-blvmj\") pod \"8d1abe95-9939-4d31-b756-321a446bc9fa\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " Jul 24 18:00:18.699134 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.698856 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-credential-socket\") pod \"8d1abe95-9939-4d31-b756-321a446bc9fa\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " Jul 24 18:00:18.699134 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.698918 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-envoy\") pod \"8d1abe95-9939-4d31-b756-321a446bc9fa\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " Jul 24 18:00:18.699134 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.698937 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d1abe95-9939-4d31-b756-321a446bc9fa" (UID: "8d1abe95-9939-4d31-b756-321a446bc9fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:00:18.699134 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.698959 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-certs\") pod \"8d1abe95-9939-4d31-b756-321a446bc9fa\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " Jul 24 18:00:18.699134 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.699020 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-data\") pod \"8d1abe95-9939-4d31-b756-321a446bc9fa\" (UID: \"8d1abe95-9939-4d31-b756-321a446bc9fa\") " Jul 24 18:00:18.699408 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.699181 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1abe95-9939-4d31-b756-321a446bc9fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d1abe95-9939-4d31-b756-321a446bc9fa" (UID: "8d1abe95-9939-4d31-b756-321a446bc9fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:00:18.699408 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.699183 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d1abe95-9939-4d31-b756-321a446bc9fa" (UID: "8d1abe95-9939-4d31-b756-321a446bc9fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:00:18.699408 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.699222 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d1abe95-9939-4d31-b756-321a446bc9fa" (UID: "8d1abe95-9939-4d31-b756-321a446bc9fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:00:18.699754 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.699419 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1abe95-9939-4d31-b756-321a446bc9fa-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:00:18.699754 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.699439 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:00:18.699754 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.699448 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:00:18.699754 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.699458 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:00:18.700288 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.700262 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d1abe95-9939-4d31-b756-321a446bc9fa" (UID: "8d1abe95-9939-4d31-b756-321a446bc9fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:00:18.701291 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.701266 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d1abe95-9939-4d31-b756-321a446bc9fa" (UID: "8d1abe95-9939-4d31-b756-321a446bc9fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:00:18.701403 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.701339 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d1abe95-9939-4d31-b756-321a446bc9fa" (UID: "8d1abe95-9939-4d31-b756-321a446bc9fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:00:18.701655 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.701636 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-kube-api-access-blvmj" (OuterVolumeSpecName: "kube-api-access-blvmj") pod "8d1abe95-9939-4d31-b756-321a446bc9fa" (UID: "8d1abe95-9939-4d31-b756-321a446bc9fa"). InnerVolumeSpecName "kube-api-access-blvmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:00:18.701714 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.701655 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d1abe95-9939-4d31-b756-321a446bc9fa" (UID: "8d1abe95-9939-4d31-b756-321a446bc9fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:00:18.800244 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.800137 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:00:18.800244 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.800175 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:00:18.800244 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.800188 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:00:18.800244 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.800198 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1abe95-9939-4d31-b756-321a446bc9fa-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:00:18.800244 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.800211 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blvmj\" (UniqueName: \"kubernetes.io/projected/8d1abe95-9939-4d31-b756-321a446bc9fa-kube-api-access-blvmj\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:00:18.846050 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.846018 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerID="a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60" exitCode=0 Jul 24 18:00:18.846252 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.846086 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" Jul 24 18:00:18.846252 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.846102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" event={"ID":"8d1abe95-9939-4d31-b756-321a446bc9fa","Type":"ContainerDied","Data":"a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60"} Jul 24 18:00:18.846252 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.846156 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b" event={"ID":"8d1abe95-9939-4d31-b756-321a446bc9fa","Type":"ContainerDied","Data":"cf226cf291a7b0b98af2f32b103a6c1626b1837b0e9cc7c6e332972ad92fe2f9"} Jul 24 18:00:18.846252 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.846177 2574 scope.go:117] "RemoveContainer" containerID="a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60" Jul 24 18:00:18.856372 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.856350 2574 scope.go:117] "RemoveContainer" containerID="39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6" Jul 24 18:00:18.865099 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.865083 2574 scope.go:117] "RemoveContainer" containerID="a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60" Jul 24 18:00:18.865367 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:00:18.865347 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60\": container with ID starting with a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60 not found: ID does not exist" containerID="a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60" Jul 24 18:00:18.865440 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.865376 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60"} err="failed to get container status \"a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60\": rpc error: code = NotFound desc = could not find container \"a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60\": container with ID starting with a607da1728992cc5a868acfd82e2f945e4b12273c1a50fd2c9d3b599f2a93d60 not found: ID does not exist" Jul 24 18:00:18.865440 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.865394 2574 scope.go:117] "RemoveContainer" containerID="39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6" Jul 24 18:00:18.865628 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:00:18.865611 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6\": container with ID starting with 39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6 not found: ID does not exist" containerID="39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6" Jul 24 18:00:18.865670 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.865636 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6"} err="failed to get container status \"39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6\": rpc error: code = NotFound desc = could not find container \"39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6\": container with ID starting with 39a3f7a858a9bcf5a29e4ffffa31f7d6fa7db61bd6b34c5f0229de4027787fa6 not found: ID does not exist" Jul 24 18:00:18.888951 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.888922 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b"] Jul 24 18:00:18.893044 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:18.893019 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0ed7b2-openshift-default-77cf6d565d-9597b"] Jul 24 18:00:19.801981 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:19.801950 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" path="/var/lib/kubelet/pods/8d1abe95-9939-4d31-b756-321a446bc9fa/volumes" Jul 24 18:00:21.951048 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:21.951015 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq"] Jul 24 18:00:21.951500 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:21.951482 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" Jul 24 18:00:21.951500 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:21.951502 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" Jul 24 18:00:21.951674 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:21.951647 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" Jul 24 18:00:21.951674 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:21.951671 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" Jul 24 18:00:21.956580 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:21.956559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:21.960154 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:21.960132 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-06ffdb-openshift-default-dockercfg-6zqkf\"" Jul 24 18:00:21.968075 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:21.968048 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq"] Jul 24 18:00:22.129455 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.129418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-data\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.129455 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.129454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb466\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-kube-api-access-lb466\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.129683 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.129504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-podinfo\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.129683 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.129522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-token\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.129683 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.129546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-envoy\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.129683 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.129615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-socket\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.129683 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.129657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istiod-ca-cert\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.129925 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.129685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-credential-socket\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.129925 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.129712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-certs\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.230621 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.230581 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-data\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.230621 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.230621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb466\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-kube-api-access-lb466\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.230917 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.230663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-podinfo\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.230917 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.230680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-token\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.230917 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.230701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-envoy\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.230917 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.230758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-socket\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.231126 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.230906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istiod-ca-cert\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.231126 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.230968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-credential-socket\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.231126 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.231063 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-data\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.231126 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.231092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-socket\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.231348 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.231134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-certs\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.231348 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.231226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-credential-socket\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.231456 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.231428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-certs\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.231643 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.231623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istiod-ca-cert\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.233182 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.233160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-envoy\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.233564 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.233543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-podinfo\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.244179 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.244148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-token\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.244278 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.244185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb466\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-kube-api-access-lb466\") pod \"e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.268400 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.268369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:22.418566 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.418541 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq"] Jul 24 18:00:22.420812 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:00:22.420779 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1283ebc4_b443_460d_b1b8_b8e1c3e62505.slice/crio-2059954d47f0f4b7e8967d20dffe7ac2fa395748813bd75ee40183b1d666520d WatchSource:0}: Error finding container 2059954d47f0f4b7e8967d20dffe7ac2fa395748813bd75ee40183b1d666520d: Status 404 returned error can't find the container with id 2059954d47f0f4b7e8967d20dffe7ac2fa395748813bd75ee40183b1d666520d Jul 24 18:00:22.423218 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.423174 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:22.423310 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.423250 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:22.423310 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.423278 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:22.873296 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.873190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" event={"ID":"1283ebc4-b443-460d-b1b8-b8e1c3e62505","Type":"ContainerStarted","Data":"9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b"} Jul 24 18:00:22.873296 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.873257 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" event={"ID":"1283ebc4-b443-460d-b1b8-b8e1c3e62505","Type":"ContainerStarted","Data":"2059954d47f0f4b7e8967d20dffe7ac2fa395748813bd75ee40183b1d666520d"} Jul 24 18:00:22.898475 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:22.898420 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" podStartSLOduration=1.898404106 podStartE2EDuration="1.898404106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:00:22.895770739 +0000 UTC m=+2579.561345376" watchObservedRunningTime="2026-07-24 18:00:22.898404106 +0000 UTC m=+2579.563978820" Jul 24 18:00:23.268574 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:23.268531 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:23.273876 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:23.273854 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:23.878885 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:23.878848 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:23.879850 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:23.879825 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:24.177353 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:24.177279 2574 scope.go:117] "RemoveContainer" containerID="9b6d29852cca106b022b0b782ae84ba309a4449220aae185460e9c84f26a684f" Jul 24 18:00:25.066997 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:25.066955 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:00:32.836280 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:32.835618 2574 patch_prober.go:28] interesting pod/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 18:00:32.836280 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:32.835704 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 18:00:32.933424 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:32.933389 2574 generic.go:356] "Generic (PLEG): container finished" podID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerID="dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf" exitCode=0 Jul 24 18:00:32.933603 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:32.933464 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" event={"ID":"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e","Type":"ContainerDied","Data":"dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf"} Jul 24 18:00:32.933919 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:32.933902 2574 scope.go:117] "RemoveContainer" containerID="dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf" Jul 24 18:00:32.935068 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:32.935039 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:32.935139 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:32.935115 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:32.935176 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:32.935149 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:33.945845 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:33.945802 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" event={"ID":"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e","Type":"ContainerStarted","Data":"00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c"} Jul 24 18:00:34.562294 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:34.562251 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:34.562713 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:34.562680 2574 patch_prober.go:28] interesting pod/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Jul 24 18:00:34.562872 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:34.562766 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Jul 24 18:00:35.562391 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:35.562352 2574 patch_prober.go:28] interesting pod/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Jul 24 18:00:35.563059 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:35.563010 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Jul 24 18:00:36.388897 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:36.388857 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:00:36.562216 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:36.562180 2574 patch_prober.go:28] interesting pod/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Jul 24 18:00:36.562521 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:36.562494 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Jul 24 18:00:37.561893 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:37.561851 2574 patch_prober.go:28] interesting pod/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Jul 24 18:00:37.562108 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:37.561947 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Jul 24 18:00:38.566913 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:38.566871 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:38.567377 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:38.567282 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:38.568030 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:38.568014 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:00:39.919921 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:39.919877 2574 patch_prober.go:28] interesting pod/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 18:00:39.920324 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:39.919943 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 18:00:41.005029 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:41.004997 2574 generic.go:356] "Generic (PLEG): container finished" podID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerID="9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b" exitCode=0 Jul 24 18:00:41.005460 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:41.005037 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" event={"ID":"1283ebc4-b443-460d-b1b8-b8e1c3e62505","Type":"ContainerDied","Data":"9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b"} Jul 24 18:00:41.005460 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:41.005432 2574 scope.go:117] "RemoveContainer" containerID="9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b" Jul 24 18:00:41.006451 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:41.006419 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:41.006530 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:41.006506 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:41.006572 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:41.006540 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:00:42.012810 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:42.012774 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" event={"ID":"1283ebc4-b443-460d-b1b8-b8e1c3e62505","Type":"ContainerStarted","Data":"c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5"} Jul 24 18:00:42.269016 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:42.268918 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:42.269204 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:42.269181 2574 patch_prober.go:28] interesting pod/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 24 18:00:42.269250 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:42.269232 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 24 18:00:43.269588 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:43.269556 2574 patch_prober.go:28] interesting pod/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 24 18:00:43.270071 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:43.269614 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 24 18:00:44.269458 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:44.269372 2574 patch_prober.go:28] interesting pod/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 24 18:00:44.269458 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:44.269429 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 24 18:00:45.269933 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:45.269893 2574 patch_prober.go:28] interesting pod/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 24 18:00:45.270370 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:45.269957 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 24 18:00:46.272830 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:46.272791 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:46.273269 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:46.273054 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:46.273649 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:46.273635 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:00:55.472060 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:00:55.472025 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:01:12.684783 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:01:12.684737 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:01:36.383044 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:01:36.383007 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:01:54.973514 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:01:54.973477 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:02:10.870847 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:02:10.870810 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:02:24.048339 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:02:24.048310 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 18:02:24.056973 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:02:24.056951 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 18:02:29.679073 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:02:29.679032 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:02:47.274292 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:02:47.274253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:03:02.296816 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:03:02.296779 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:03:18.079965 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:03:18.079920 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:03:28.716994 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:03:28.716953 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:04:24.240493 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:04:24.240406 2574 scope.go:117] "RemoveContainer" containerID="c6ed9baed08856524e59198724e857a62ca4e2ed2cbe5e9327e71f5de6929380" Jul 24 18:04:28.384977 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:04:28.384940 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:04:47.594183 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:04:47.594141 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:05:10.471012 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:05:10.470973 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:05:22.978321 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:05:22.978231 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:05:24.265455 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:05:24.265368 2574 scope.go:117] "RemoveContainer" containerID="af5edb7ca5ab8987feb9ae95b9598b601cbc0893c31a9e84e9d81df3eb423d92" Jul 24 18:05:24.274849 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:05:24.274817 2574 scope.go:117] "RemoveContainer" containerID="99059317625b1e28c21c86a05ecc610ae38a1729015a869525160fb5c74b0c16" Jul 24 18:05:48.881614 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:05:48.881578 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:06:02.772105 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:02.772065 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:06:17.789013 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:17.788975 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:06:24.300296 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:24.300215 2574 scope.go:117] "RemoveContainer" containerID="a6f872d0f6ba639c806cc94da7afaf660fbb46fa9d630b95f0913a89cc1235f9" Jul 24 18:06:24.309231 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:24.309207 2574 scope.go:117] "RemoveContainer" containerID="6c4fd61d17b26d3286db4d3907eec1eac54fd8e15bada267e0c42a25d9743cb0" Jul 24 18:06:27.887263 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:27.887231 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7571-140019849705904"] Jul 24 18:06:27.887679 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:27.887639 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" Jul 24 18:06:27.887679 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:27.887650 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1abe95-9939-4d31-b756-321a446bc9fa" containerName="istio-proxy" Jul 24 18:06:27.892380 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:27.892354 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849705904" Jul 24 18:06:27.895565 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:27.895539 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-swkk2\"" Jul 24 18:06:27.896383 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:27.896357 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019849705904"] Jul 24 18:06:28.021885 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:28.021844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxf2q\" (UniqueName: \"kubernetes.io/projected/13096eee-af27-4db8-b343-a51127e61804-kube-api-access-vxf2q\") pod \"test-curl-7571-140019849705904\" (UID: \"13096eee-af27-4db8-b343-a51127e61804\") " pod="opendatahub/test-curl-7571-140019849705904" Jul 24 18:06:28.122896 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:28.122850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxf2q\" (UniqueName: \"kubernetes.io/projected/13096eee-af27-4db8-b343-a51127e61804-kube-api-access-vxf2q\") pod \"test-curl-7571-140019849705904\" (UID: \"13096eee-af27-4db8-b343-a51127e61804\") " pod="opendatahub/test-curl-7571-140019849705904" Jul 24 18:06:28.132601 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:28.132576 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxf2q\" (UniqueName: \"kubernetes.io/projected/13096eee-af27-4db8-b343-a51127e61804-kube-api-access-vxf2q\") pod \"test-curl-7571-140019849705904\" (UID: \"13096eee-af27-4db8-b343-a51127e61804\") " pod="opendatahub/test-curl-7571-140019849705904" Jul 24 18:06:28.202470 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:28.202387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849705904" Jul 24 18:06:28.344795 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:28.344764 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019849705904"] Jul 24 18:06:28.349395 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:06:28.349365 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13096eee_af27_4db8_b343_a51127e61804.slice/crio-687d495be2c9f169feb4fbf24e3c2ae5e2516f74decd1414f7f3d40af6f19dc2 WatchSource:0}: Error finding container 687d495be2c9f169feb4fbf24e3c2ae5e2516f74decd1414f7f3d40af6f19dc2: Status 404 returned error can't find the container with id 687d495be2c9f169feb4fbf24e3c2ae5e2516f74decd1414f7f3d40af6f19dc2 Jul 24 18:06:28.350804 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:28.350783 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 18:06:29.098952 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:29.098921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019849705904" event={"ID":"13096eee-af27-4db8-b343-a51127e61804","Type":"ContainerStarted","Data":"687d495be2c9f169feb4fbf24e3c2ae5e2516f74decd1414f7f3d40af6f19dc2"} Jul 24 18:06:30.106011 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:30.105968 2574 generic.go:356] "Generic (PLEG): container finished" podID="13096eee-af27-4db8-b343-a51127e61804" containerID="e16b4d0a2675b7646076c3e773c0ac8fbcf15392f47e9ff9501eb14ad9e13ed8" exitCode=0 Jul 24 18:06:30.106537 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:30.106040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019849705904" event={"ID":"13096eee-af27-4db8-b343-a51127e61804","Type":"ContainerDied","Data":"e16b4d0a2675b7646076c3e773c0ac8fbcf15392f47e9ff9501eb14ad9e13ed8"} Jul 24 18:06:31.257514 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.257486 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849705904" Jul 24 18:06:31.298129 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.298102 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7571-140019849705904_13096eee-af27-4db8-b343-a51127e61804/test-curl-7571-140019849705904/0.log" Jul 24 18:06:31.325511 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.325466 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7571-140019849705904"] Jul 24 18:06:31.331650 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.331623 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7571-140019849705904"] Jul 24 18:06:31.354635 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.354608 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxf2q\" (UniqueName: \"kubernetes.io/projected/13096eee-af27-4db8-b343-a51127e61804-kube-api-access-vxf2q\") pod \"13096eee-af27-4db8-b343-a51127e61804\" (UID: \"13096eee-af27-4db8-b343-a51127e61804\") " Jul 24 18:06:31.356968 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.356947 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13096eee-af27-4db8-b343-a51127e61804-kube-api-access-vxf2q" (OuterVolumeSpecName: "kube-api-access-vxf2q") pod "13096eee-af27-4db8-b343-a51127e61804" (UID: "13096eee-af27-4db8-b343-a51127e61804"). InnerVolumeSpecName "kube-api-access-vxf2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:06:31.455667 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.455637 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxf2q\" (UniqueName: \"kubernetes.io/projected/13096eee-af27-4db8-b343-a51127e61804-kube-api-access-vxf2q\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:06:31.465492 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.465461 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7571-140019849705776"] Jul 24 18:06:31.465884 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.465868 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13096eee-af27-4db8-b343-a51127e61804" containerName="test-curl-7571-140019849705904" Jul 24 18:06:31.465935 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.465887 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="13096eee-af27-4db8-b343-a51127e61804" containerName="test-curl-7571-140019849705904" Jul 24 18:06:31.465971 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.465965 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="13096eee-af27-4db8-b343-a51127e61804" containerName="test-curl-7571-140019849705904" Jul 24 18:06:31.469087 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.469071 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849705776" Jul 24 18:06:31.477246 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.477217 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019849705776"] Jul 24 18:06:31.556394 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.556360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplp6\" (UniqueName: \"kubernetes.io/projected/060e3eec-8f46-477a-aa26-777af0de7e73-kube-api-access-zplp6\") pod \"test-curl-7571-140019849705776\" (UID: \"060e3eec-8f46-477a-aa26-777af0de7e73\") " pod="opendatahub/test-curl-7571-140019849705776" Jul 24 18:06:31.657196 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.657082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplp6\" (UniqueName: \"kubernetes.io/projected/060e3eec-8f46-477a-aa26-777af0de7e73-kube-api-access-zplp6\") pod \"test-curl-7571-140019849705776\" (UID: \"060e3eec-8f46-477a-aa26-777af0de7e73\") " pod="opendatahub/test-curl-7571-140019849705776" Jul 24 18:06:31.666688 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.666657 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplp6\" (UniqueName: \"kubernetes.io/projected/060e3eec-8f46-477a-aa26-777af0de7e73-kube-api-access-zplp6\") pod \"test-curl-7571-140019849705776\" (UID: \"060e3eec-8f46-477a-aa26-777af0de7e73\") " pod="opendatahub/test-curl-7571-140019849705776" Jul 24 18:06:31.779115 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.779080 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849705776" Jul 24 18:06:31.803938 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.803908 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13096eee-af27-4db8-b343-a51127e61804" path="/var/lib/kubelet/pods/13096eee-af27-4db8-b343-a51127e61804/volumes" Jul 24 18:06:31.922310 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:31.922287 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019849705776"] Jul 24 18:06:31.923609 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:06:31.923588 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod060e3eec_8f46_477a_aa26_777af0de7e73.slice/crio-ceea4eb97ad97d13eed679edcf972de7069ecc529fc02f8bac0759aa1cddba1e WatchSource:0}: Error finding container ceea4eb97ad97d13eed679edcf972de7069ecc529fc02f8bac0759aa1cddba1e: Status 404 returned error can't find the container with id ceea4eb97ad97d13eed679edcf972de7069ecc529fc02f8bac0759aa1cddba1e Jul 24 18:06:32.118628 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:32.118602 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849705904" Jul 24 18:06:32.118836 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:32.118637 2574 scope.go:117] "RemoveContainer" containerID="e16b4d0a2675b7646076c3e773c0ac8fbcf15392f47e9ff9501eb14ad9e13ed8" Jul 24 18:06:32.120032 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:32.120012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019849705776" event={"ID":"060e3eec-8f46-477a-aa26-777af0de7e73","Type":"ContainerStarted","Data":"ceea4eb97ad97d13eed679edcf972de7069ecc529fc02f8bac0759aa1cddba1e"} Jul 24 18:06:32.679808 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:32.679767 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:06:33.127966 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:33.127931 2574 generic.go:356] "Generic (PLEG): container finished" podID="060e3eec-8f46-477a-aa26-777af0de7e73" containerID="dd1c4625f2aace2d251c559b96525d5325edbb1531c6b09aac6894933aba1b99" exitCode=0 Jul 24 18:06:33.128176 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:33.128010 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019849705776" event={"ID":"060e3eec-8f46-477a-aa26-777af0de7e73","Type":"ContainerDied","Data":"dd1c4625f2aace2d251c559b96525d5325edbb1531c6b09aac6894933aba1b99"} Jul 24 18:06:34.274873 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.274844 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849705776" Jul 24 18:06:34.300799 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.300767 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7571-140019849705776_060e3eec-8f46-477a-aa26-777af0de7e73/test-curl-7571-140019849705776/0.log" Jul 24 18:06:34.327857 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.327826 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7571-140019849705776"] Jul 24 18:06:34.330255 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.330235 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7571-140019849705776"] Jul 24 18:06:34.384454 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.384419 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zplp6\" (UniqueName: \"kubernetes.io/projected/060e3eec-8f46-477a-aa26-777af0de7e73-kube-api-access-zplp6\") pod \"060e3eec-8f46-477a-aa26-777af0de7e73\" (UID: \"060e3eec-8f46-477a-aa26-777af0de7e73\") " Jul 24 18:06:34.386836 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.386772 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060e3eec-8f46-477a-aa26-777af0de7e73-kube-api-access-zplp6" (OuterVolumeSpecName: "kube-api-access-zplp6") pod "060e3eec-8f46-477a-aa26-777af0de7e73" (UID: "060e3eec-8f46-477a-aa26-777af0de7e73"). InnerVolumeSpecName "kube-api-access-zplp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:06:34.485607 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.485574 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zplp6\" (UniqueName: \"kubernetes.io/projected/060e3eec-8f46-477a-aa26-777af0de7e73-kube-api-access-zplp6\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:06:34.503203 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.503165 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:34.503571 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.503558 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060e3eec-8f46-477a-aa26-777af0de7e73" containerName="test-curl-7571-140019849705776" Jul 24 18:06:34.503626 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.503573 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="060e3eec-8f46-477a-aa26-777af0de7e73" containerName="test-curl-7571-140019849705776" Jul 24 18:06:34.503670 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.503661 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="060e3eec-8f46-477a-aa26-777af0de7e73" containerName="test-curl-7571-140019849705776" Jul 24 18:06:34.506638 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.506617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:34.514444 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.514423 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:34.586914 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.586874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2x8l\" (UniqueName: \"kubernetes.io/projected/92802e0b-7094-408e-bb1e-f45662a0f10e-kube-api-access-n2x8l\") pod \"test-curl-7571-140019851954192\" (UID: \"92802e0b-7094-408e-bb1e-f45662a0f10e\") " pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:34.688507 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.688418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2x8l\" (UniqueName: \"kubernetes.io/projected/92802e0b-7094-408e-bb1e-f45662a0f10e-kube-api-access-n2x8l\") pod \"test-curl-7571-140019851954192\" (UID: \"92802e0b-7094-408e-bb1e-f45662a0f10e\") " pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:34.698755 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.698713 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2x8l\" (UniqueName: \"kubernetes.io/projected/92802e0b-7094-408e-bb1e-f45662a0f10e-kube-api-access-n2x8l\") pod \"test-curl-7571-140019851954192\" (UID: \"92802e0b-7094-408e-bb1e-f45662a0f10e\") " pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:34.817388 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.817351 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:34.960881 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:34.960855 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:34.962314 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:06:34.962288 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92802e0b_7094_408e_bb1e_f45662a0f10e.slice/crio-eae42d71f9d627cb8c7b15b09ddcb5704840ffd81703e73f8d2449254f9d206a WatchSource:0}: Error finding container eae42d71f9d627cb8c7b15b09ddcb5704840ffd81703e73f8d2449254f9d206a: Status 404 returned error can't find the container with id eae42d71f9d627cb8c7b15b09ddcb5704840ffd81703e73f8d2449254f9d206a Jul 24 18:06:35.141294 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:35.141251 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019851954192" event={"ID":"92802e0b-7094-408e-bb1e-f45662a0f10e","Type":"ContainerStarted","Data":"eae42d71f9d627cb8c7b15b09ddcb5704840ffd81703e73f8d2449254f9d206a"} Jul 24 18:06:35.142441 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:35.142423 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceea4eb97ad97d13eed679edcf972de7069ecc529fc02f8bac0759aa1cddba1e" Jul 24 18:06:35.142441 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:35.142437 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849705776" Jul 24 18:06:35.153251 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:06:35.153223 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7571-140019849705776\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="060e3eec-8f46-477a-aa26-777af0de7e73" pod="opendatahub/test-curl-7571-140019849705776" Jul 24 18:06:35.801771 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:35.801709 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060e3eec-8f46-477a-aa26-777af0de7e73" path="/var/lib/kubelet/pods/060e3eec-8f46-477a-aa26-777af0de7e73/volumes" Jul 24 18:06:36.149631 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:36.149545 2574 generic.go:356] "Generic (PLEG): container finished" podID="92802e0b-7094-408e-bb1e-f45662a0f10e" containerID="a12399f5223677a4256c5e30a0c23f9b30caf48f855c2eceac6de8bc4ce386aa" exitCode=0 Jul 24 18:06:36.149815 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:36.149635 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019851954192" event={"ID":"92802e0b-7094-408e-bb1e-f45662a0f10e","Type":"ContainerDied","Data":"a12399f5223677a4256c5e30a0c23f9b30caf48f855c2eceac6de8bc4ce386aa"} Jul 24 18:06:37.298297 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.298272 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:37.324492 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.324468 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7571-140019851954192_92802e0b-7094-408e-bb1e-f45662a0f10e/test-curl-7571-140019851954192/0.log" Jul 24 18:06:37.351756 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.351704 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:37.356370 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.356341 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:37.412712 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.412608 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2x8l\" (UniqueName: \"kubernetes.io/projected/92802e0b-7094-408e-bb1e-f45662a0f10e-kube-api-access-n2x8l\") pod \"92802e0b-7094-408e-bb1e-f45662a0f10e\" (UID: \"92802e0b-7094-408e-bb1e-f45662a0f10e\") " Jul 24 18:06:37.415072 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.415040 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92802e0b-7094-408e-bb1e-f45662a0f10e-kube-api-access-n2x8l" (OuterVolumeSpecName: "kube-api-access-n2x8l") pod "92802e0b-7094-408e-bb1e-f45662a0f10e" (UID: "92802e0b-7094-408e-bb1e-f45662a0f10e"). InnerVolumeSpecName "kube-api-access-n2x8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:06:37.485697 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.485659 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:37.486182 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.486165 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92802e0b-7094-408e-bb1e-f45662a0f10e" containerName="test-curl-7571-140019851954192" Jul 24 18:06:37.486234 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.486184 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="92802e0b-7094-408e-bb1e-f45662a0f10e" containerName="test-curl-7571-140019851954192" Jul 24 18:06:37.486267 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.486249 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="92802e0b-7094-408e-bb1e-f45662a0f10e" containerName="test-curl-7571-140019851954192" Jul 24 18:06:37.489245 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.489229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:37.498938 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.498916 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:37.514336 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.514312 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2x8l\" (UniqueName: \"kubernetes.io/projected/92802e0b-7094-408e-bb1e-f45662a0f10e-kube-api-access-n2x8l\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:06:37.615072 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.615030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xfs\" (UniqueName: \"kubernetes.io/projected/ceec7109-e529-4a32-810d-018d6dafd418-kube-api-access-q8xfs\") pod \"test-curl-7571-140019851954192\" (UID: \"ceec7109-e529-4a32-810d-018d6dafd418\") " pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:37.716123 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.716088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xfs\" (UniqueName: \"kubernetes.io/projected/ceec7109-e529-4a32-810d-018d6dafd418-kube-api-access-q8xfs\") pod \"test-curl-7571-140019851954192\" (UID: \"ceec7109-e529-4a32-810d-018d6dafd418\") " pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:37.727160 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.727138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xfs\" (UniqueName: \"kubernetes.io/projected/ceec7109-e529-4a32-810d-018d6dafd418-kube-api-access-q8xfs\") pod \"test-curl-7571-140019851954192\" (UID: \"ceec7109-e529-4a32-810d-018d6dafd418\") " pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:37.799512 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.799474 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:37.803359 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.803331 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92802e0b-7094-408e-bb1e-f45662a0f10e" path="/var/lib/kubelet/pods/92802e0b-7094-408e-bb1e-f45662a0f10e/volumes" Jul 24 18:06:37.951788 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:37.951756 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:37.952947 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:06:37.952923 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceec7109_e529_4a32_810d_018d6dafd418.slice/crio-0959c42fc7bcfc7a8b5a46acb92ec72b851b5269555311377d356cddb7a47b64 WatchSource:0}: Error finding container 0959c42fc7bcfc7a8b5a46acb92ec72b851b5269555311377d356cddb7a47b64: Status 404 returned error can't find the container with id 0959c42fc7bcfc7a8b5a46acb92ec72b851b5269555311377d356cddb7a47b64 Jul 24 18:06:38.164253 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:38.164224 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:38.164460 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:38.164281 2574 scope.go:117] "RemoveContainer" containerID="a12399f5223677a4256c5e30a0c23f9b30caf48f855c2eceac6de8bc4ce386aa" Jul 24 18:06:38.165764 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:38.165720 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019851954192" event={"ID":"ceec7109-e529-4a32-810d-018d6dafd418","Type":"ContainerStarted","Data":"0959c42fc7bcfc7a8b5a46acb92ec72b851b5269555311377d356cddb7a47b64"} Jul 24 18:06:39.173290 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:39.173254 2574 generic.go:356] "Generic (PLEG): container finished" podID="ceec7109-e529-4a32-810d-018d6dafd418" containerID="ee717e479a6675d3fe3dedf6d777361043e10e9d528d1b78558c91e8d24bde0e" exitCode=0 Jul 24 18:06:39.173740 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:39.173343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019851954192" event={"ID":"ceec7109-e529-4a32-810d-018d6dafd418","Type":"ContainerDied","Data":"ee717e479a6675d3fe3dedf6d777361043e10e9d528d1b78558c91e8d24bde0e"} Jul 24 18:06:40.327641 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.327615 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:40.350007 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.349976 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7571-140019851954192_ceec7109-e529-4a32-810d-018d6dafd418/test-curl-7571-140019851954192/0.log" Jul 24 18:06:40.377165 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.377134 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:40.381465 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.381438 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7571-140019851954192"] Jul 24 18:06:40.442538 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.442449 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8xfs\" (UniqueName: \"kubernetes.io/projected/ceec7109-e529-4a32-810d-018d6dafd418-kube-api-access-q8xfs\") pod \"ceec7109-e529-4a32-810d-018d6dafd418\" (UID: \"ceec7109-e529-4a32-810d-018d6dafd418\") " Jul 24 18:06:40.444793 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.444768 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceec7109-e529-4a32-810d-018d6dafd418-kube-api-access-q8xfs" (OuterVolumeSpecName: "kube-api-access-q8xfs") pod "ceec7109-e529-4a32-810d-018d6dafd418" (UID: "ceec7109-e529-4a32-810d-018d6dafd418"). InnerVolumeSpecName "kube-api-access-q8xfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:06:40.514218 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.514182 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:40.514640 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.514625 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceec7109-e529-4a32-810d-018d6dafd418" containerName="test-curl-7571-140019851954192" Jul 24 18:06:40.514682 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.514644 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceec7109-e529-4a32-810d-018d6dafd418" containerName="test-curl-7571-140019851954192" Jul 24 18:06:40.514742 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.514715 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceec7109-e529-4a32-810d-018d6dafd418" containerName="test-curl-7571-140019851954192" Jul 24 18:06:40.517788 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.517772 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:40.525277 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.525242 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:40.543650 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.543611 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8xfs\" (UniqueName: \"kubernetes.io/projected/ceec7109-e529-4a32-810d-018d6dafd418-kube-api-access-q8xfs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:06:40.644336 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.644293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jff9\" (UniqueName: \"kubernetes.io/projected/2127680c-ad20-4f8e-9b03-4ca497d32cba-kube-api-access-5jff9\") pod \"test-curl-7571-140019849725392\" (UID: \"2127680c-ad20-4f8e-9b03-4ca497d32cba\") " pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:40.745744 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.745699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jff9\" (UniqueName: \"kubernetes.io/projected/2127680c-ad20-4f8e-9b03-4ca497d32cba-kube-api-access-5jff9\") pod \"test-curl-7571-140019849725392\" (UID: \"2127680c-ad20-4f8e-9b03-4ca497d32cba\") " pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:40.756209 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.756186 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jff9\" (UniqueName: \"kubernetes.io/projected/2127680c-ad20-4f8e-9b03-4ca497d32cba-kube-api-access-5jff9\") pod \"test-curl-7571-140019849725392\" (UID: \"2127680c-ad20-4f8e-9b03-4ca497d32cba\") " pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:40.828216 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.828176 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:40.971316 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:40.971293 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:41.189265 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:41.189230 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0959c42fc7bcfc7a8b5a46acb92ec72b851b5269555311377d356cddb7a47b64" Jul 24 18:06:41.189265 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:41.189241 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:41.190675 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:41.190645 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019849725392" event={"ID":"2127680c-ad20-4f8e-9b03-4ca497d32cba","Type":"ContainerStarted","Data":"4c43e1018ac0adfdf67700eeca778d9281045f99d0df2789409fe2892ece2ba1"} Jul 24 18:06:41.201435 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:06:41.201406 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7571-140019851954192\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="ceec7109-e529-4a32-810d-018d6dafd418" pod="opendatahub/test-curl-7571-140019851954192" Jul 24 18:06:41.802380 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:41.802340 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceec7109-e529-4a32-810d-018d6dafd418" path="/var/lib/kubelet/pods/ceec7109-e529-4a32-810d-018d6dafd418/volumes" Jul 24 18:06:42.198465 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:42.198381 2574 generic.go:356] "Generic (PLEG): container finished" podID="2127680c-ad20-4f8e-9b03-4ca497d32cba" containerID="c00e587fccec4f0f59068ad4225d9fabb4f1994d4a9eb62daa84035943ba43b3" exitCode=0 Jul 24 18:06:42.198603 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:42.198473 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019849725392" event={"ID":"2127680c-ad20-4f8e-9b03-4ca497d32cba","Type":"ContainerDied","Data":"c00e587fccec4f0f59068ad4225d9fabb4f1994d4a9eb62daa84035943ba43b3"} Jul 24 18:06:43.347743 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.347707 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:43.369119 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.369067 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7571-140019849725392_2127680c-ad20-4f8e-9b03-4ca497d32cba/test-curl-7571-140019849725392/0.log" Jul 24 18:06:43.394235 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.394195 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:43.398951 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.398921 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:43.473505 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.473479 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jff9\" (UniqueName: \"kubernetes.io/projected/2127680c-ad20-4f8e-9b03-4ca497d32cba-kube-api-access-5jff9\") pod \"2127680c-ad20-4f8e-9b03-4ca497d32cba\" (UID: \"2127680c-ad20-4f8e-9b03-4ca497d32cba\") " Jul 24 18:06:43.475790 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.475765 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2127680c-ad20-4f8e-9b03-4ca497d32cba-kube-api-access-5jff9" (OuterVolumeSpecName: "kube-api-access-5jff9") pod "2127680c-ad20-4f8e-9b03-4ca497d32cba" (UID: "2127680c-ad20-4f8e-9b03-4ca497d32cba"). InnerVolumeSpecName "kube-api-access-5jff9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:06:43.528523 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.528490 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:43.528923 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.528907 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2127680c-ad20-4f8e-9b03-4ca497d32cba" containerName="test-curl-7571-140019849725392" Jul 24 18:06:43.528972 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.528925 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2127680c-ad20-4f8e-9b03-4ca497d32cba" containerName="test-curl-7571-140019849725392" Jul 24 18:06:43.529015 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.529006 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2127680c-ad20-4f8e-9b03-4ca497d32cba" containerName="test-curl-7571-140019849725392" Jul 24 18:06:43.532089 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.532074 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:43.539226 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.539197 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:43.574396 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.574361 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jff9\" (UniqueName: \"kubernetes.io/projected/2127680c-ad20-4f8e-9b03-4ca497d32cba-kube-api-access-5jff9\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:06:43.675654 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.675614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9bw\" (UniqueName: \"kubernetes.io/projected/1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb-kube-api-access-wq9bw\") pod \"test-curl-7571-140019849725392\" (UID: \"1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb\") " pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:43.776640 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.776528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9bw\" (UniqueName: \"kubernetes.io/projected/1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb-kube-api-access-wq9bw\") pod \"test-curl-7571-140019849725392\" (UID: \"1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb\") " pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:43.786187 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.786147 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9bw\" (UniqueName: \"kubernetes.io/projected/1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb-kube-api-access-wq9bw\") pod \"test-curl-7571-140019849725392\" (UID: \"1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb\") " pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:43.801508 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.801475 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2127680c-ad20-4f8e-9b03-4ca497d32cba" path="/var/lib/kubelet/pods/2127680c-ad20-4f8e-9b03-4ca497d32cba/volumes" Jul 24 18:06:43.842826 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.842785 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:43.985086 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:43.985061 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:43.986996 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:06:43.986965 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d9a4f80_8dbe_4eef_85dd_65761cc1e4cb.slice/crio-a3669018cfc7c35f7eab3a012085eb1c174ff8decba151bd4ffdc1349b3f7d48 WatchSource:0}: Error finding container a3669018cfc7c35f7eab3a012085eb1c174ff8decba151bd4ffdc1349b3f7d48: Status 404 returned error can't find the container with id a3669018cfc7c35f7eab3a012085eb1c174ff8decba151bd4ffdc1349b3f7d48 Jul 24 18:06:44.212380 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:44.212345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019849725392" event={"ID":"1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb","Type":"ContainerStarted","Data":"a3669018cfc7c35f7eab3a012085eb1c174ff8decba151bd4ffdc1349b3f7d48"} Jul 24 18:06:44.213664 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:44.213643 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:44.213801 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:44.213672 2574 scope.go:117] "RemoveContainer" containerID="c00e587fccec4f0f59068ad4225d9fabb4f1994d4a9eb62daa84035943ba43b3" Jul 24 18:06:45.222432 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:45.222391 2574 generic.go:356] "Generic (PLEG): container finished" podID="1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb" containerID="df2d4a15241302f4afc9ba1b90098cad4d9b6f010bf3a39556c1bf6cc63e7c20" exitCode=0 Jul 24 18:06:45.222847 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:45.222483 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7571-140019849725392" event={"ID":"1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb","Type":"ContainerDied","Data":"df2d4a15241302f4afc9ba1b90098cad4d9b6f010bf3a39556c1bf6cc63e7c20"} Jul 24 18:06:46.371147 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:46.371125 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:46.394768 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:46.394718 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7571-140019849725392_1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb/test-curl-7571-140019849725392/0.log" Jul 24 18:06:46.421359 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:46.421313 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:46.427887 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:46.427855 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7571-140019849725392"] Jul 24 18:06:46.505473 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:46.505372 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq9bw\" (UniqueName: \"kubernetes.io/projected/1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb-kube-api-access-wq9bw\") pod \"1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb\" (UID: \"1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb\") " Jul 24 18:06:46.507527 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:46.507504 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb-kube-api-access-wq9bw" (OuterVolumeSpecName: "kube-api-access-wq9bw") pod "1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb" (UID: "1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb"). InnerVolumeSpecName "kube-api-access-wq9bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:06:46.606682 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:46.606641 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9bw\" (UniqueName: \"kubernetes.io/projected/1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb-kube-api-access-wq9bw\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:06:47.237197 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.237168 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:47.237354 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.237205 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3669018cfc7c35f7eab3a012085eb1c174ff8decba151bd4ffdc1349b3f7d48" Jul 24 18:06:47.249819 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:06:47.249788 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7571-140019849725392\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:47.346502 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.346466 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4"] Jul 24 18:06:47.346934 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.346919 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb" containerName="test-curl-7571-140019849725392" Jul 24 18:06:47.346934 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.346934 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb" containerName="test-curl-7571-140019849725392" Jul 24 18:06:47.347035 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.347018 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb" containerName="test-curl-7571-140019849725392" Jul 24 18:06:47.350127 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.350111 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.353892 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:06:47.353860 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7571-140019849725392\" is forbidden: User \"system:node:ip-10-0-130-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-58.ec2.internal' and this object" podUID="1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb" pod="opendatahub/test-curl-7571-140019849725392" Jul 24 18:06:47.354196 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.354178 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-737ef6-openshift-default-dockercfg-vkdzv\"" Jul 24 18:06:47.368718 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.368691 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4"] Jul 24 18:06:47.517860 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.517720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69127808-cd49-4772-b959-991f3f84521f-istiod-ca-cert\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.517860 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.517819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-envoy\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.517860 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.517846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-credential-socket\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.517860 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.517863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69127808-cd49-4772-b959-991f3f84521f-istio-podinfo\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.518381 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.517900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-istio-token\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.518381 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.518010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-certs\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.518381 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.518049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn8d4\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-kube-api-access-vn8d4\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.518381 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.518123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-socket\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.518381 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.518188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-data\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.619818 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.619777 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-certs\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620033 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.619831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn8d4\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-kube-api-access-vn8d4\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620033 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.619876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-socket\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620033 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.619939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-data\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620033 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.619990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69127808-cd49-4772-b959-991f3f84521f-istiod-ca-cert\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620250 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.620033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-envoy\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620250 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.620081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-credential-socket\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620250 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.620181 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69127808-cd49-4772-b959-991f3f84521f-istio-podinfo\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620250 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.620229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-istio-token\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620435 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.620311 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-certs\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620435 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.620336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-socket\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620510 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.620477 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-credential-socket\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620706 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.620660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-data\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.620706 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.620683 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69127808-cd49-4772-b959-991f3f84521f-istiod-ca-cert\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.622616 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.622593 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-envoy\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.622813 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.622795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69127808-cd49-4772-b959-991f3f84521f-istio-podinfo\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.631888 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.631863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn8d4\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-kube-api-access-vn8d4\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.632659 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.632637 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-istio-token\") pod \"e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.660678 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.660635 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:47.806702 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.806667 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb" path="/var/lib/kubelet/pods/1d9a4f80-8dbe-4eef-85dd-65761cc1e4cb/volumes" Jul 24 18:06:47.850276 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.850253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4"] Jul 24 18:06:47.852183 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:06:47.852159 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69127808_cd49_4772_b959_991f3f84521f.slice/crio-3ed5fd5146fcaa8812e502c3e47818afc5f825581133cb3dcefd6459a26e3fdf WatchSource:0}: Error finding container 3ed5fd5146fcaa8812e502c3e47818afc5f825581133cb3dcefd6459a26e3fdf: Status 404 returned error can't find the container with id 3ed5fd5146fcaa8812e502c3e47818afc5f825581133cb3dcefd6459a26e3fdf Jul 24 18:06:47.854465 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.854432 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:06:47.854531 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.854503 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:06:47.854571 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:47.854532 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:06:48.245925 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:48.245889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" event={"ID":"69127808-cd49-4772-b959-991f3f84521f","Type":"ContainerStarted","Data":"cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c"} Jul 24 18:06:48.246096 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:48.245943 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" event={"ID":"69127808-cd49-4772-b959-991f3f84521f","Type":"ContainerStarted","Data":"3ed5fd5146fcaa8812e502c3e47818afc5f825581133cb3dcefd6459a26e3fdf"} Jul 24 18:06:48.283146 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:48.283092 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" podStartSLOduration=1.283075607 podStartE2EDuration="1.283075607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:06:48.279597191 +0000 UTC m=+2964.945171863" watchObservedRunningTime="2026-07-24 18:06:48.283075607 +0000 UTC m=+2964.948650244" Jul 24 18:06:48.660934 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:48.660840 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:48.666156 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:48.666128 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:49.252152 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:49.252111 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:49.253166 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:49.253148 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:06:59.792198 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:06:59.792157 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:07:06.135616 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.135578 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk"] Jul 24 18:07:06.140087 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.140058 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.143511 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.143447 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-53e30e-openshift-default-dockercfg-lz7jl\"" Jul 24 18:07:06.160202 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.160172 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk"] Jul 24 18:07:06.216133 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.216094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-podinfo\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.216346 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.216155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-token\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.216346 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.216300 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-socket\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.216480 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.216343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-data\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.216480 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.216436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-certs\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.216480 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.216467 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-envoy\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.216649 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.216523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2txp\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-kube-api-access-g2txp\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.216649 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.216568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-credential-socket\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.216649 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.216592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istiod-ca-cert\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.317841 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.317808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-certs\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.317841 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.317844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-envoy\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318094 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.317875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2txp\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-kube-api-access-g2txp\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318094 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.317900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-credential-socket\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318094 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istiod-ca-cert\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318271 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-podinfo\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318271 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-token\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318271 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-socket\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318424 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-data\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318424 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318334 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-credential-socket\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318545 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-certs\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318609 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318594 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-data\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318816 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318792 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-socket\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.318908 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.318835 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istiod-ca-cert\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.320897 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.320866 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-envoy\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.320897 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.320888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-podinfo\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.327293 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.327265 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-token\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.327963 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.327942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2txp\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-kube-api-access-g2txp\") pod \"e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.452496 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.452398 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:06.623281 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.623251 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk"] Jul 24 18:07:06.624955 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:07:06.624926 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f8dc49_3969_44b7_bfb8_5c6f668c4059.slice/crio-be23953de361a4e81ba50d21d8fee18c470b1dc4cc140dc7394644a10b7f11f2 WatchSource:0}: Error finding container be23953de361a4e81ba50d21d8fee18c470b1dc4cc140dc7394644a10b7f11f2: Status 404 returned error can't find the container with id be23953de361a4e81ba50d21d8fee18c470b1dc4cc140dc7394644a10b7f11f2 Jul 24 18:07:06.627470 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.627428 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:07:06.627564 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.627519 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:07:06.627604 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:06.627562 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:07:07.386928 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:07.386892 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" event={"ID":"c0f8dc49-3969-44b7-bfb8-5c6f668c4059","Type":"ContainerStarted","Data":"f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30"} Jul 24 18:07:07.388503 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:07.386975 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" event={"ID":"c0f8dc49-3969-44b7-bfb8-5c6f668c4059","Type":"ContainerStarted","Data":"be23953de361a4e81ba50d21d8fee18c470b1dc4cc140dc7394644a10b7f11f2"} Jul 24 18:07:07.416085 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:07.416014 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" podStartSLOduration=1.415993911 podStartE2EDuration="1.415993911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:07:07.411841131 +0000 UTC m=+2984.077415769" watchObservedRunningTime="2026-07-24 18:07:07.415993911 +0000 UTC m=+2984.081568550" Jul 24 18:07:07.453604 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:07.453130 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:07.458610 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:07.458585 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:08.395419 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:08.395388 2574 generic.go:356] "Generic (PLEG): container finished" podID="69127808-cd49-4772-b959-991f3f84521f" containerID="cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c" exitCode=0 Jul 24 18:07:08.395858 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:08.395466 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" event={"ID":"69127808-cd49-4772-b959-991f3f84521f","Type":"ContainerDied","Data":"cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c"} Jul 24 18:07:08.395858 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:08.395703 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:08.395988 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:08.395972 2574 scope.go:117] "RemoveContainer" containerID="cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c" Jul 24 18:07:08.396857 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:08.396773 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:08.397143 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:08.397110 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:07:08.397235 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:08.397199 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:07:08.397296 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:08.397246 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:07:09.403978 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:09.403938 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" event={"ID":"69127808-cd49-4772-b959-991f3f84521f","Type":"ContainerStarted","Data":"d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8"} Jul 24 18:07:09.661113 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:09.661016 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:07:09.661296 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:09.661276 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Jul 24 18:07:09.661360 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:09.661335 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Jul 24 18:07:10.661860 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:10.661824 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Jul 24 18:07:10.662339 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:10.661887 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Jul 24 18:07:11.661793 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:11.661743 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Jul 24 18:07:11.661986 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:11.661821 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Jul 24 18:07:12.661378 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:12.661344 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Jul 24 18:07:12.661558 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:12.661412 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Jul 24 18:07:13.665542 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:13.665516 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:07:13.666060 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:13.665952 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:07:13.666811 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:13.666794 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:07:19.009958 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:19.009924 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:07:24.090819 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:24.090787 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 18:07:24.101236 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:24.101209 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 18:07:27.529461 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:27.529430 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerID="f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30" exitCode=0 Jul 24 18:07:27.529961 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:27.529510 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" event={"ID":"c0f8dc49-3969-44b7-bfb8-5c6f668c4059","Type":"ContainerDied","Data":"f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30"} Jul 24 18:07:27.530030 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:27.529971 2574 scope.go:117] "RemoveContainer" containerID="f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30" Jul 24 18:07:27.531060 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:27.531031 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:07:27.531140 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:27.531100 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:07:27.531140 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:27.531133 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 18:07:28.541371 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:28.541333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" event={"ID":"c0f8dc49-3969-44b7-bfb8-5c6f668c4059","Type":"ContainerStarted","Data":"1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83"} Jul 24 18:07:29.453654 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:29.453616 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:29.453960 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:29.453926 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Jul 24 18:07:29.454056 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:29.454016 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Jul 24 18:07:30.454099 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:30.453599 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Jul 24 18:07:30.454099 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:30.453678 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Jul 24 18:07:31.453337 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:31.453297 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Jul 24 18:07:31.453549 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:31.453369 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Jul 24 18:07:32.453327 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:32.453280 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Jul 24 18:07:32.453888 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:32.453350 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Jul 24 18:07:33.457000 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:33.456967 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:33.457402 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:33.457261 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:33.458101 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:33.458083 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:07:49.690939 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:07:49.690900 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:08:05.276003 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:05.275965 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:08:25.879816 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:25.879777 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:08:35.556392 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:35.556351 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl"] Jul 24 18:08:35.560004 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:35.559987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" Jul 24 18:08:35.563271 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:35.563253 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 18:08:35.569680 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:35.569659 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl"] Jul 24 18:08:35.712043 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:35.712000 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl\" (UID: \"7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" Jul 24 18:08:35.813033 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:35.812933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl\" (UID: \"7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" Jul 24 18:08:35.813558 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:35.813537 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl\" (UID: \"7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" Jul 24 18:08:35.872637 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:35.872597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" Jul 24 18:08:36.022259 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:36.022227 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl"] Jul 24 18:08:36.022940 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:08:36.022917 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fefc1de_c6cb_4cfd_9f05_09aa4ea24ebb.slice/crio-aeee3cf358e19981b0c3d4122b94f1810d5d60e0baeac591564dc7d2c9a521da WatchSource:0}: Error finding container aeee3cf358e19981b0c3d4122b94f1810d5d60e0baeac591564dc7d2c9a521da: Status 404 returned error can't find the container with id aeee3cf358e19981b0c3d4122b94f1810d5d60e0baeac591564dc7d2c9a521da Jul 24 18:08:36.974538 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:36.974500 2574 generic.go:356] "Generic (PLEG): container finished" podID="7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb" containerID="5533319bc57bac7641709f6feda36ef2558c371276393c52470f79db132fc36d" exitCode=0 Jul 24 18:08:36.974992 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:36.974543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" event={"ID":"7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb","Type":"ContainerDied","Data":"5533319bc57bac7641709f6feda36ef2558c371276393c52470f79db132fc36d"} Jul 24 18:08:36.974992 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:36.974598 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" event={"ID":"7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb","Type":"ContainerStarted","Data":"aeee3cf358e19981b0c3d4122b94f1810d5d60e0baeac591564dc7d2c9a521da"} Jul 24 18:08:38.133082 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:38.133054 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" Jul 24 18:08:38.238637 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:38.238547 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb-maas-api-service-ca\") pod \"7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb\" (UID: \"7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb\") " Jul 24 18:08:38.238990 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:38.238964 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb" (UID: "7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:08:38.339505 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:38.339471 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:08:38.988516 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:38.988487 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" Jul 24 18:08:38.988672 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:38.988521 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl" event={"ID":"7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb","Type":"ContainerDied","Data":"aeee3cf358e19981b0c3d4122b94f1810d5d60e0baeac591564dc7d2c9a521da"} Jul 24 18:08:38.988672 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:38.988573 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeee3cf358e19981b0c3d4122b94f1810d5d60e0baeac591564dc7d2c9a521da" Jul 24 18:08:50.655368 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:50.655327 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl"] Jul 24 18:08:50.657993 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:50.657967 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fb4092-sjvcl"] Jul 24 18:08:50.975881 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:50.975846 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:08:51.801645 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:08:51.801613 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb" path="/var/lib/kubelet/pods/7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb/volumes" Jul 24 18:09:01.344544 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.344507 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l"] Jul 24 18:09:01.345050 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.344988 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb" containerName="revoke-keys" Jul 24 18:09:01.345050 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.345002 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb" containerName="revoke-keys" Jul 24 18:09:01.345127 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.345066 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fefc1de-c6cb-4cfd-9f05-09aa4ea24ebb" containerName="revoke-keys" Jul 24 18:09:01.349396 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.349374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" Jul 24 18:09:01.352648 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.352621 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 18:09:01.357245 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.357215 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l"] Jul 24 18:09:01.472267 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.472223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed7faa-27b6-4bcc-9290-03a254c7989f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l\" (UID: \"51ed7faa-27b6-4bcc-9290-03a254c7989f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" Jul 24 18:09:01.573410 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.573370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed7faa-27b6-4bcc-9290-03a254c7989f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l\" (UID: \"51ed7faa-27b6-4bcc-9290-03a254c7989f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" Jul 24 18:09:01.574057 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.574038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed7faa-27b6-4bcc-9290-03a254c7989f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l\" (UID: \"51ed7faa-27b6-4bcc-9290-03a254c7989f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" Jul 24 18:09:01.661665 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.661576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" Jul 24 18:09:01.807242 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:01.807214 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l"] Jul 24 18:09:01.808456 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:09:01.808432 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ed7faa_27b6_4bcc_9290_03a254c7989f.slice/crio-45731323049b2b775300df6235f87e444066fd428c452b44c87886d48333ef63 WatchSource:0}: Error finding container 45731323049b2b775300df6235f87e444066fd428c452b44c87886d48333ef63: Status 404 returned error can't find the container with id 45731323049b2b775300df6235f87e444066fd428c452b44c87886d48333ef63 Jul 24 18:09:02.139116 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:02.139085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" event={"ID":"51ed7faa-27b6-4bcc-9290-03a254c7989f","Type":"ContainerStarted","Data":"45731323049b2b775300df6235f87e444066fd428c452b44c87886d48333ef63"} Jul 24 18:09:03.147020 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:03.146986 2574 generic.go:356] "Generic (PLEG): container finished" podID="51ed7faa-27b6-4bcc-9290-03a254c7989f" containerID="4ee5ac7606b19fab0df57d17e871952ed9b29b7893e6304af3443a4084ecf068" exitCode=0 Jul 24 18:09:03.147445 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:03.147077 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" event={"ID":"51ed7faa-27b6-4bcc-9290-03a254c7989f","Type":"ContainerDied","Data":"4ee5ac7606b19fab0df57d17e871952ed9b29b7893e6304af3443a4084ecf068"} Jul 24 18:09:04.305281 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:04.305251 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" Jul 24 18:09:04.400745 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:04.400695 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed7faa-27b6-4bcc-9290-03a254c7989f-maas-api-service-ca\") pod \"51ed7faa-27b6-4bcc-9290-03a254c7989f\" (UID: \"51ed7faa-27b6-4bcc-9290-03a254c7989f\") " Jul 24 18:09:04.401076 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:04.401048 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ed7faa-27b6-4bcc-9290-03a254c7989f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51ed7faa-27b6-4bcc-9290-03a254c7989f" (UID: "51ed7faa-27b6-4bcc-9290-03a254c7989f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:09:04.502213 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:04.502177 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed7faa-27b6-4bcc-9290-03a254c7989f-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:09:05.163567 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:05.163532 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" Jul 24 18:09:05.163567 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:05.163562 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l" event={"ID":"51ed7faa-27b6-4bcc-9290-03a254c7989f","Type":"ContainerDied","Data":"45731323049b2b775300df6235f87e444066fd428c452b44c87886d48333ef63"} Jul 24 18:09:05.163828 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:05.163610 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45731323049b2b775300df6235f87e444066fd428c452b44c87886d48333ef63" Jul 24 18:09:16.573503 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:16.573467 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:09:21.463630 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:21.463596 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l"] Jul 24 18:09:21.471319 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:21.471285 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-737ef6-k9r8l"] Jul 24 18:09:21.802374 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:21.802289 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ed7faa-27b6-4bcc-9290-03a254c7989f" path="/var/lib/kubelet/pods/51ed7faa-27b6-4bcc-9290-03a254c7989f/volumes" Jul 24 18:09:26.883693 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:09:26.883648 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:10:17.007720 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:17.007683 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4"] Jul 24 18:10:17.008371 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:17.007956 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" containerID="cri-o://d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8" gracePeriod=30 Jul 24 18:10:22.265205 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.265176 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:10:22.373800 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.373680 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-istio-token\") pod \"69127808-cd49-4772-b959-991f3f84521f\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " Jul 24 18:10:22.373800 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.373800 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-socket\") pod \"69127808-cd49-4772-b959-991f3f84521f\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " Jul 24 18:10:22.374169 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.373826 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-credential-socket\") pod \"69127808-cd49-4772-b959-991f3f84521f\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " Jul 24 18:10:22.374169 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.373868 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69127808-cd49-4772-b959-991f3f84521f-istio-podinfo\") pod \"69127808-cd49-4772-b959-991f3f84521f\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " Jul 24 18:10:22.374169 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.373895 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-data\") pod \"69127808-cd49-4772-b959-991f3f84521f\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " Jul 24 18:10:22.374169 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.373937 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn8d4\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-kube-api-access-vn8d4\") pod \"69127808-cd49-4772-b959-991f3f84521f\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " Jul 24 18:10:22.374169 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.373974 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-envoy\") pod \"69127808-cd49-4772-b959-991f3f84521f\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " Jul 24 18:10:22.374169 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.374070 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-certs\") pod \"69127808-cd49-4772-b959-991f3f84521f\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " Jul 24 18:10:22.374169 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.374115 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69127808-cd49-4772-b959-991f3f84521f" (UID: "69127808-cd49-4772-b959-991f3f84521f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:22.374534 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.374171 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69127808-cd49-4772-b959-991f3f84521f" (UID: "69127808-cd49-4772-b959-991f3f84521f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:22.374534 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.374377 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69127808-cd49-4772-b959-991f3f84521f" (UID: "69127808-cd49-4772-b959-991f3f84521f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:22.374534 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.374402 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69127808-cd49-4772-b959-991f3f84521f-istiod-ca-cert\") pod \"69127808-cd49-4772-b959-991f3f84521f\" (UID: \"69127808-cd49-4772-b959-991f3f84521f\") " Jul 24 18:10:22.374997 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.374970 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69127808-cd49-4772-b959-991f3f84521f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69127808-cd49-4772-b959-991f3f84521f" (UID: "69127808-cd49-4772-b959-991f3f84521f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:10:22.375077 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.375011 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:22.375077 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.375032 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:22.375077 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.375048 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:22.375247 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.375125 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-data" (OuterVolumeSpecName: "istio-data") pod "69127808-cd49-4772-b959-991f3f84521f" (UID: "69127808-cd49-4772-b959-991f3f84521f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:22.376321 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.376293 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69127808-cd49-4772-b959-991f3f84521f" (UID: "69127808-cd49-4772-b959-991f3f84521f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:22.376450 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.376358 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69127808-cd49-4772-b959-991f3f84521f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69127808-cd49-4772-b959-991f3f84521f" (UID: "69127808-cd49-4772-b959-991f3f84521f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:10:22.376450 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.376409 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-kube-api-access-vn8d4" (OuterVolumeSpecName: "kube-api-access-vn8d4") pod "69127808-cd49-4772-b959-991f3f84521f" (UID: "69127808-cd49-4772-b959-991f3f84521f"). InnerVolumeSpecName "kube-api-access-vn8d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:10:22.376450 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.376422 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-istio-token" (OuterVolumeSpecName: "istio-token") pod "69127808-cd49-4772-b959-991f3f84521f" (UID: "69127808-cd49-4772-b959-991f3f84521f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:10:22.475944 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.475905 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:22.475944 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.475936 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69127808-cd49-4772-b959-991f3f84521f-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:22.475944 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.475946 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:22.475944 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.475955 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn8d4\" (UniqueName: \"kubernetes.io/projected/69127808-cd49-4772-b959-991f3f84521f-kube-api-access-vn8d4\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:22.476228 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.475964 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69127808-cd49-4772-b959-991f3f84521f-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:22.476228 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.475974 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69127808-cd49-4772-b959-991f3f84521f-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:22.654775 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.654669 2574 generic.go:356] "Generic (PLEG): container finished" podID="69127808-cd49-4772-b959-991f3f84521f" containerID="d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8" exitCode=0 Jul 24 18:10:22.654775 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.654719 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" event={"ID":"69127808-cd49-4772-b959-991f3f84521f","Type":"ContainerDied","Data":"d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8"} Jul 24 18:10:22.654775 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.654755 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" Jul 24 18:10:22.655045 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.654791 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4" event={"ID":"69127808-cd49-4772-b959-991f3f84521f","Type":"ContainerDied","Data":"3ed5fd5146fcaa8812e502c3e47818afc5f825581133cb3dcefd6459a26e3fdf"} Jul 24 18:10:22.655045 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.654816 2574 scope.go:117] "RemoveContainer" containerID="d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8" Jul 24 18:10:22.665583 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.665561 2574 scope.go:117] "RemoveContainer" containerID="cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c" Jul 24 18:10:22.675421 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.675402 2574 scope.go:117] "RemoveContainer" containerID="d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8" Jul 24 18:10:22.675753 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:10:22.675704 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8\": container with ID starting with d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8 not found: ID does not exist" containerID="d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8" Jul 24 18:10:22.675883 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.675761 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8"} err="failed to get container status \"d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8\": rpc error: code = NotFound desc = could not find container \"d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8\": container with ID starting with d2918b699c327f877dfc2c39790df4f6b745d71600da9016760955f572bf08c8 not found: ID does not exist" Jul 24 18:10:22.675883 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.675783 2574 scope.go:117] "RemoveContainer" containerID="cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c" Jul 24 18:10:22.676113 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:10:22.676088 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c\": container with ID starting with cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c not found: ID does not exist" containerID="cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c" Jul 24 18:10:22.676204 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.676124 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c"} err="failed to get container status \"cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c\": rpc error: code = NotFound desc = could not find container \"cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c\": container with ID starting with cd69c60dd67afe8a4f5038ab3790f73a7eb1093cb71e736b6d9b05180a310e3c not found: ID does not exist" Jul 24 18:10:22.679677 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.679652 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4"] Jul 24 18:10:22.684964 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:22.684939 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-737ef6-openshift-default-5446c9c85d-6xtf4"] Jul 24 18:10:23.802695 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:23.802659 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69127808-cd49-4772-b959-991f3f84521f" path="/var/lib/kubelet/pods/69127808-cd49-4772-b959-991f3f84521f/volumes" Jul 24 18:10:31.467449 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:31.467416 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:10:39.459690 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:39.459654 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk"] Jul 24 18:10:39.460138 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:39.459876 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" containerID="cri-o://1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83" gracePeriod=30 Jul 24 18:10:44.720767 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.720687 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:10:44.788961 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.788932 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerID="1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83" exitCode=0 Jul 24 18:10:44.789158 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.788986 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" Jul 24 18:10:44.789158 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.789026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" event={"ID":"c0f8dc49-3969-44b7-bfb8-5c6f668c4059","Type":"ContainerDied","Data":"1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83"} Jul 24 18:10:44.789158 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.789083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk" event={"ID":"c0f8dc49-3969-44b7-bfb8-5c6f668c4059","Type":"ContainerDied","Data":"be23953de361a4e81ba50d21d8fee18c470b1dc4cc140dc7394644a10b7f11f2"} Jul 24 18:10:44.789158 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.789107 2574 scope.go:117] "RemoveContainer" containerID="1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83" Jul 24 18:10:44.799388 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.799369 2574 scope.go:117] "RemoveContainer" containerID="f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30" Jul 24 18:10:44.808241 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.808220 2574 scope.go:117] "RemoveContainer" containerID="1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83" Jul 24 18:10:44.808508 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:10:44.808488 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83\": container with ID starting with 1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83 not found: ID does not exist" containerID="1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83" Jul 24 18:10:44.808557 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.808520 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83"} err="failed to get container status \"1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83\": rpc error: code = NotFound desc = could not find container \"1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83\": container with ID starting with 1f9986e4f9523cb7e05650a80c35f0651aa65a6e1c3d473d2c08b4f659d65d83 not found: ID does not exist" Jul 24 18:10:44.808557 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.808539 2574 scope.go:117] "RemoveContainer" containerID="f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30" Jul 24 18:10:44.808822 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:10:44.808804 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30\": container with ID starting with f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30 not found: ID does not exist" containerID="f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30" Jul 24 18:10:44.808889 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.808832 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30"} err="failed to get container status \"f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30\": rpc error: code = NotFound desc = could not find container \"f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30\": container with ID starting with f248fa4641145d0aed589bbbf6c81afc6d3a5ba126faad067d1a8d706b462a30 not found: ID does not exist" Jul 24 18:10:44.896274 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896235 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-data\") pod \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " Jul 24 18:10:44.896460 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896315 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-socket\") pod \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " Jul 24 18:10:44.896460 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896340 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-certs\") pod \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " Jul 24 18:10:44.896460 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896406 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-podinfo\") pod \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " Jul 24 18:10:44.896460 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896438 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istiod-ca-cert\") pod \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " Jul 24 18:10:44.896661 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896463 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-credential-socket\") pod \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " Jul 24 18:10:44.896661 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896493 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2txp\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-kube-api-access-g2txp\") pod \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " Jul 24 18:10:44.896661 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896597 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0f8dc49-3969-44b7-bfb8-5c6f668c4059" (UID: "c0f8dc49-3969-44b7-bfb8-5c6f668c4059"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:44.896857 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896699 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-envoy\") pod \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " Jul 24 18:10:44.896857 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896750 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0f8dc49-3969-44b7-bfb8-5c6f668c4059" (UID: "c0f8dc49-3969-44b7-bfb8-5c6f668c4059"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:44.896857 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896791 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-token\") pod \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\" (UID: \"c0f8dc49-3969-44b7-bfb8-5c6f668c4059\") " Jul 24 18:10:44.897019 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896865 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0f8dc49-3969-44b7-bfb8-5c6f668c4059" (UID: "c0f8dc49-3969-44b7-bfb8-5c6f668c4059"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:44.897019 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.896891 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0f8dc49-3969-44b7-bfb8-5c6f668c4059" (UID: "c0f8dc49-3969-44b7-bfb8-5c6f668c4059"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:10:44.897135 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.897099 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0f8dc49-3969-44b7-bfb8-5c6f668c4059" (UID: "c0f8dc49-3969-44b7-bfb8-5c6f668c4059"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:44.897135 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.897122 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:44.897245 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.897136 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:44.897245 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.897147 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:44.897245 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.897156 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:44.898989 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.898959 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0f8dc49-3969-44b7-bfb8-5c6f668c4059" (UID: "c0f8dc49-3969-44b7-bfb8-5c6f668c4059"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:10:44.898989 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.898964 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0f8dc49-3969-44b7-bfb8-5c6f668c4059" (UID: "c0f8dc49-3969-44b7-bfb8-5c6f668c4059"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:44.898989 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.898976 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0f8dc49-3969-44b7-bfb8-5c6f668c4059" (UID: "c0f8dc49-3969-44b7-bfb8-5c6f668c4059"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:10:44.899345 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.899324 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-kube-api-access-g2txp" (OuterVolumeSpecName: "kube-api-access-g2txp") pod "c0f8dc49-3969-44b7-bfb8-5c6f668c4059" (UID: "c0f8dc49-3969-44b7-bfb8-5c6f668c4059"). InnerVolumeSpecName "kube-api-access-g2txp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:10:44.997811 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.997686 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:44.997811 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.997720 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:44.997811 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.997761 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:44.997811 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.997770 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2txp\" (UniqueName: \"kubernetes.io/projected/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-kube-api-access-g2txp\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:44.997811 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:44.997781 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f8dc49-3969-44b7-bfb8-5c6f668c4059-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:10:45.114692 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:45.114659 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk"] Jul 24 18:10:45.117084 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:45.117060 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-53e30e-openshift-default-56bd4cf6db-8fsqk"] Jul 24 18:10:45.806929 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:45.806895 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" path="/var/lib/kubelet/pods/c0f8dc49-3969-44b7-bfb8-5c6f668c4059/volumes" Jul 24 18:10:54.375625 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:10:54.375586 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:11:02.085034 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.084997 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz"] Jul 24 18:11:02.085469 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.085248 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" containerID="cri-o://00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c" gracePeriod=30 Jul 24 18:11:02.604438 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604399 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9"] Jul 24 18:11:02.604848 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604833 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" Jul 24 18:11:02.604902 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604849 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" Jul 24 18:11:02.604902 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604857 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" Jul 24 18:11:02.604902 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604862 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" Jul 24 18:11:02.604902 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604877 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" Jul 24 18:11:02.604902 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604883 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" Jul 24 18:11:02.604902 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604892 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ed7faa-27b6-4bcc-9290-03a254c7989f" containerName="revoke-keys" Jul 24 18:11:02.604902 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604897 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ed7faa-27b6-4bcc-9290-03a254c7989f" containerName="revoke-keys" Jul 24 18:11:02.605136 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604907 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" Jul 24 18:11:02.605136 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604913 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" Jul 24 18:11:02.605136 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604972 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ed7faa-27b6-4bcc-9290-03a254c7989f" containerName="revoke-keys" Jul 24 18:11:02.605136 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604981 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" Jul 24 18:11:02.605136 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604986 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="69127808-cd49-4772-b959-991f3f84521f" containerName="istio-proxy" Jul 24 18:11:02.605136 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604992 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" Jul 24 18:11:02.605136 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.604998 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f8dc49-3969-44b7-bfb8-5c6f668c4059" containerName="istio-proxy" Jul 24 18:11:02.609459 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.609440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" Jul 24 18:11:02.614426 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.614405 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c25tj\"" Jul 24 18:11:02.630975 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.630935 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9"] Jul 24 18:11:02.674026 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.673990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b66f1-5eea-4a4a-8b5a-5b7828facd3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9\" (UID: \"044b66f1-5eea-4a4a-8b5a-5b7828facd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" Jul 24 18:11:02.774743 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.774697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b66f1-5eea-4a4a-8b5a-5b7828facd3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9\" (UID: \"044b66f1-5eea-4a4a-8b5a-5b7828facd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" Jul 24 18:11:02.775343 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.775322 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b66f1-5eea-4a4a-8b5a-5b7828facd3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9\" (UID: \"044b66f1-5eea-4a4a-8b5a-5b7828facd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" Jul 24 18:11:02.920051 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:02.919959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" Jul 24 18:11:03.276269 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:03.276248 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9"] Jul 24 18:11:03.279617 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:11:03.279585 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044b66f1_5eea_4a4a_8b5a_5b7828facd3e.slice/crio-c37fbaf516b1aac8ce9f433bf82a761e59b15ab4a69b0958aa35c12de0c6a10a WatchSource:0}: Error finding container c37fbaf516b1aac8ce9f433bf82a761e59b15ab4a69b0958aa35c12de0c6a10a: Status 404 returned error can't find the container with id c37fbaf516b1aac8ce9f433bf82a761e59b15ab4a69b0958aa35c12de0c6a10a Jul 24 18:11:03.905690 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:03.905589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" event={"ID":"044b66f1-5eea-4a4a-8b5a-5b7828facd3e","Type":"ContainerStarted","Data":"5fb5b4a9104e3f80cdfbc7a4476687c8377c8379c3cda023212ba3829127f8f4"} Jul 24 18:11:03.905690 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:03.905657 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" event={"ID":"044b66f1-5eea-4a4a-8b5a-5b7828facd3e","Type":"ContainerStarted","Data":"c37fbaf516b1aac8ce9f433bf82a761e59b15ab4a69b0958aa35c12de0c6a10a"} Jul 24 18:11:03.925785 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:03.925708 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" podStartSLOduration=1.925693314 podStartE2EDuration="1.925693314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:11:03.924805757 +0000 UTC m=+3220.590380398" watchObservedRunningTime="2026-07-24 18:11:03.925693314 +0000 UTC m=+3220.591267951" Jul 24 18:11:04.913675 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:04.913637 2574 generic.go:356] "Generic (PLEG): container finished" podID="044b66f1-5eea-4a4a-8b5a-5b7828facd3e" containerID="5fb5b4a9104e3f80cdfbc7a4476687c8377c8379c3cda023212ba3829127f8f4" exitCode=0 Jul 24 18:11:04.914110 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:04.913683 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" event={"ID":"044b66f1-5eea-4a4a-8b5a-5b7828facd3e","Type":"ContainerDied","Data":"5fb5b4a9104e3f80cdfbc7a4476687c8377c8379c3cda023212ba3829127f8f4"} Jul 24 18:11:06.065624 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:06.065593 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" Jul 24 18:11:06.206380 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:06.206279 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b66f1-5eea-4a4a-8b5a-5b7828facd3e-maas-api-service-ca\") pod \"044b66f1-5eea-4a4a-8b5a-5b7828facd3e\" (UID: \"044b66f1-5eea-4a4a-8b5a-5b7828facd3e\") " Jul 24 18:11:06.206667 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:06.206642 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044b66f1-5eea-4a4a-8b5a-5b7828facd3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "044b66f1-5eea-4a4a-8b5a-5b7828facd3e" (UID: "044b66f1-5eea-4a4a-8b5a-5b7828facd3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:11:06.307693 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:06.307653 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b66f1-5eea-4a4a-8b5a-5b7828facd3e-maas-api-service-ca\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:06.926669 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:06.926635 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" Jul 24 18:11:06.926893 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:06.926672 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9" event={"ID":"044b66f1-5eea-4a4a-8b5a-5b7828facd3e","Type":"ContainerDied","Data":"c37fbaf516b1aac8ce9f433bf82a761e59b15ab4a69b0958aa35c12de0c6a10a"} Jul 24 18:11:06.926893 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:06.926744 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c37fbaf516b1aac8ce9f433bf82a761e59b15ab4a69b0958aa35c12de0c6a10a" Jul 24 18:11:07.340061 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.340034 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:11:07.419119 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419085 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rnxh\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-kube-api-access-5rnxh\") pod \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " Jul 24 18:11:07.419335 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419144 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-podinfo\") pod \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " Jul 24 18:11:07.419335 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419168 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-socket\") pod \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " Jul 24 18:11:07.419335 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419194 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-credential-socket\") pod \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " Jul 24 18:11:07.419335 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419229 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-token\") pod \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " Jul 24 18:11:07.419335 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419249 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istiod-ca-cert\") pod \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " Jul 24 18:11:07.419335 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419275 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-envoy\") pod \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " Jul 24 18:11:07.419667 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419351 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-data\") pod \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " Jul 24 18:11:07.419667 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419383 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-certs\") pod \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\" (UID: \"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e\") " Jul 24 18:11:07.419667 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419614 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" (UID: "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:07.419874 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419753 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:07.419874 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419764 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" (UID: "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:11:07.420142 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.419841 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" (UID: "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:07.420142 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.420067 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" (UID: "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:07.420400 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.420379 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" (UID: "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:07.421571 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.421546 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" (UID: "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:11:07.421707 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.421689 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" (UID: "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:07.421962 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.421939 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-kube-api-access-5rnxh" (OuterVolumeSpecName: "kube-api-access-5rnxh") pod "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" (UID: "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e"). InnerVolumeSpecName "kube-api-access-5rnxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:11:07.422025 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.421977 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" (UID: "6603c3a2-5079-4bf6-bf0a-2f0863a9b51e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:11:07.520803 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.520771 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:07.520803 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.520799 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:07.520803 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.520810 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:07.521062 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.520820 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:07.521062 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.520828 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:07.521062 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.520835 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:07.521062 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.520842 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:07.521062 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.520850 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rnxh\" (UniqueName: \"kubernetes.io/projected/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e-kube-api-access-5rnxh\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:07.935519 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.935487 2574 generic.go:356] "Generic (PLEG): container finished" podID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerID="00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c" exitCode=0 Jul 24 18:11:07.935758 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.935550 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" Jul 24 18:11:07.935758 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.935575 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" event={"ID":"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e","Type":"ContainerDied","Data":"00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c"} Jul 24 18:11:07.935758 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.935630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz" event={"ID":"6603c3a2-5079-4bf6-bf0a-2f0863a9b51e","Type":"ContainerDied","Data":"fb02418bd8f0c6244d8d9d57c4ddb9f2e20a009e0f1f7059270ad86a66eb9b8b"} Jul 24 18:11:07.935758 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.935653 2574 scope.go:117] "RemoveContainer" containerID="00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c" Jul 24 18:11:07.945279 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.945261 2574 scope.go:117] "RemoveContainer" containerID="dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf" Jul 24 18:11:07.954195 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.954176 2574 scope.go:117] "RemoveContainer" containerID="00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c" Jul 24 18:11:07.954493 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:11:07.954472 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c\": container with ID starting with 00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c not found: ID does not exist" containerID="00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c" Jul 24 18:11:07.954549 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.954503 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c"} err="failed to get container status \"00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c\": rpc error: code = NotFound desc = could not find container \"00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c\": container with ID starting with 00cf9071362998d59630912524beea05ddf77bdfa3ab41f75114f8c25b69633c not found: ID does not exist" Jul 24 18:11:07.954549 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.954521 2574 scope.go:117] "RemoveContainer" containerID="dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf" Jul 24 18:11:07.954855 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:11:07.954788 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf\": container with ID starting with dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf not found: ID does not exist" containerID="dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf" Jul 24 18:11:07.954855 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.954811 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf"} err="failed to get container status \"dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf\": rpc error: code = NotFound desc = could not find container \"dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf\": container with ID starting with dce74d95d9091c4aac238fd63fba2968f7332b7e8cb4f9797b943c4d68ded0bf not found: ID does not exist" Jul 24 18:11:07.961551 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.961526 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz"] Jul 24 18:11:07.970465 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:07.970435 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-257940-openshift-default-845d57f676-rhnlz"] Jul 24 18:11:09.802212 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:09.802179 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" path="/var/lib/kubelet/pods/6603c3a2-5079-4bf6-bf0a-2f0863a9b51e/volumes" Jul 24 18:11:16.068296 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:16.068207 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9rth9"] Jul 24 18:11:17.737681 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:17.737645 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9"] Jul 24 18:11:17.749463 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:17.749428 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-06ffdb-shzg9"] Jul 24 18:11:17.801453 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:17.801419 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="044b66f1-5eea-4a4a-8b5a-5b7828facd3e" path="/var/lib/kubelet/pods/044b66f1-5eea-4a4a-8b5a-5b7828facd3e/volumes" Jul 24 18:11:24.651753 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:24.651640 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq"] Jul 24 18:11:24.652159 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:24.651875 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" containerID="cri-o://c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5" gracePeriod=30 Jul 24 18:11:29.913622 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.913594 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:11:29.932653 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932572 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-certs\") pod \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " Jul 24 18:11:29.932835 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932660 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-podinfo\") pod \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " Jul 24 18:11:29.932835 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932697 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-credential-socket\") pod \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " Jul 24 18:11:29.932835 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932762 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-token\") pod \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " Jul 24 18:11:29.932835 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932786 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-socket\") pod \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " Jul 24 18:11:29.932835 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932807 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istiod-ca-cert\") pod \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " Jul 24 18:11:29.933098 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932854 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-envoy\") pod \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " Jul 24 18:11:29.933098 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb466\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-kube-api-access-lb466\") pod \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " Jul 24 18:11:29.933098 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932895 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1283ebc4-b443-460d-b1b8-b8e1c3e62505" (UID: "1283ebc4-b443-460d-b1b8-b8e1c3e62505"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:29.933098 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.932937 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-data\") pod \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\" (UID: \"1283ebc4-b443-460d-b1b8-b8e1c3e62505\") " Jul 24 18:11:29.933098 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.933018 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1283ebc4-b443-460d-b1b8-b8e1c3e62505" (UID: "1283ebc4-b443-460d-b1b8-b8e1c3e62505"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:29.933366 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.933088 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1283ebc4-b443-460d-b1b8-b8e1c3e62505" (UID: "1283ebc4-b443-460d-b1b8-b8e1c3e62505"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:29.933366 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.933283 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-credential-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:29.933366 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.933306 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-socket\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:29.933366 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.933319 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-workload-certs\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:29.933663 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.933617 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1283ebc4-b443-460d-b1b8-b8e1c3e62505" (UID: "1283ebc4-b443-460d-b1b8-b8e1c3e62505"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:11:29.933775 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.933744 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-data" (OuterVolumeSpecName: "istio-data") pod "1283ebc4-b443-460d-b1b8-b8e1c3e62505" (UID: "1283ebc4-b443-460d-b1b8-b8e1c3e62505"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:29.935251 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.935224 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-kube-api-access-lb466" (OuterVolumeSpecName: "kube-api-access-lb466") pod "1283ebc4-b443-460d-b1b8-b8e1c3e62505" (UID: "1283ebc4-b443-460d-b1b8-b8e1c3e62505"). InnerVolumeSpecName "kube-api-access-lb466". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:11:29.935671 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.935649 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1283ebc4-b443-460d-b1b8-b8e1c3e62505" (UID: "1283ebc4-b443-460d-b1b8-b8e1c3e62505"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:11:29.935671 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.935653 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1283ebc4-b443-460d-b1b8-b8e1c3e62505" (UID: "1283ebc4-b443-460d-b1b8-b8e1c3e62505"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:11:29.935671 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:29.935658 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-token" (OuterVolumeSpecName: "istio-token") pod "1283ebc4-b443-460d-b1b8-b8e1c3e62505" (UID: "1283ebc4-b443-460d-b1b8-b8e1c3e62505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:11:30.034792 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.034740 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-podinfo\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:30.034792 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.034787 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-token\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:30.034792 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.034796 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istiod-ca-cert\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:30.034792 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.034804 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-envoy\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:30.034792 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.034812 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb466\" (UniqueName: \"kubernetes.io/projected/1283ebc4-b443-460d-b1b8-b8e1c3e62505-kube-api-access-lb466\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:30.035135 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.034821 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1283ebc4-b443-460d-b1b8-b8e1c3e62505-istio-data\") on node \"ip-10-0-130-58.ec2.internal\" DevicePath \"\"" Jul 24 18:11:30.070221 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.070189 2574 generic.go:356] "Generic (PLEG): container finished" podID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerID="c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5" exitCode=0 Jul 24 18:11:30.070401 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.070221 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" event={"ID":"1283ebc4-b443-460d-b1b8-b8e1c3e62505","Type":"ContainerDied","Data":"c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5"} Jul 24 18:11:30.070401 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.070246 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" Jul 24 18:11:30.070401 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.070293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq" event={"ID":"1283ebc4-b443-460d-b1b8-b8e1c3e62505","Type":"ContainerDied","Data":"2059954d47f0f4b7e8967d20dffe7ac2fa395748813bd75ee40183b1d666520d"} Jul 24 18:11:30.070401 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.070327 2574 scope.go:117] "RemoveContainer" containerID="c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5" Jul 24 18:11:30.080487 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.080470 2574 scope.go:117] "RemoveContainer" containerID="9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b" Jul 24 18:11:30.089966 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.089939 2574 scope.go:117] "RemoveContainer" containerID="c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5" Jul 24 18:11:30.090267 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:11:30.090247 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5\": container with ID starting with c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5 not found: ID does not exist" containerID="c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5" Jul 24 18:11:30.090352 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.090275 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5"} err="failed to get container status \"c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5\": rpc error: code = NotFound desc = could not find container \"c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5\": container with ID starting with c5c491223c159f4ceac35c13bc1ab3985c5e5d273d106b7fe628698819eec7f5 not found: ID does not exist" Jul 24 18:11:30.090352 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.090295 2574 scope.go:117] "RemoveContainer" containerID="9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b" Jul 24 18:11:30.090591 ip-10-0-130-58 kubenswrapper[2574]: E0724 18:11:30.090568 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b\": container with ID starting with 9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b not found: ID does not exist" containerID="9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b" Jul 24 18:11:30.090653 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.090595 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b"} err="failed to get container status \"9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b\": rpc error: code = NotFound desc = could not find container \"9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b\": container with ID starting with 9d6b898dc1ad3b793074ea3f87d05e2382d26bc0ada8d298b3e9f4fb83cfff9b not found: ID does not exist" Jul 24 18:11:30.096773 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.096743 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq"] Jul 24 18:11:30.110337 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:30.110301 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-06ffdb-openshift-default-86567d889b-mgjcq"] Jul 24 18:11:31.447903 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:31.447874 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kbd5r_84aabd4e-2a07-48e9-a3d5-1591954a1408/manager/0.log" Jul 24 18:11:31.591689 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:31.591649 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7s2g8_5562ae78-dee0-4b9b-a693-626e0863dbf0/manager/0.log" Jul 24 18:11:31.801529 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:31.801492 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" path="/var/lib/kubelet/pods/1283ebc4-b443-460d-b1b8-b8e1c3e62505/volumes" Jul 24 18:11:32.121652 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:32.121571 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8574785b6-phgvq_6bbe55c5-07d5-4f94-a80b-15449eb8ca14/copy-manifests/0.log" Jul 24 18:11:32.145202 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:32.145166 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8574785b6-phgvq_6bbe55c5-07d5-4f94-a80b-15449eb8ca14/manager/0.log" Jul 24 18:11:32.947803 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:32.947772 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748585-n9r89_c1f8f502-62de-407d-9d45-84f7d2c192e0/cleanup/0.log" Jul 24 18:11:33.096359 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:33.096320 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748600-r5g2q_0d673fe5-02f8-4a18-8628-db7d478e221d/cleanup/0.log" Jul 24 18:11:33.227634 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:33.227589 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qlxrs_e79aff8c-138d-453f-9470-4036151fb467/postgres/0.log" Jul 24 18:11:34.121334 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.121309 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8_400c4845-05ff-4d9d-8a9a-a35b388ad2f6/util/0.log" Jul 24 18:11:34.129707 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.129660 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8_400c4845-05ff-4d9d-8a9a-a35b388ad2f6/pull/0.log" Jul 24 18:11:34.137916 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.137890 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8_400c4845-05ff-4d9d-8a9a-a35b388ad2f6/extract/0.log" Jul 24 18:11:34.273924 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.273891 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm_674513fc-7288-40ee-9a40-2747ac8f5852/util/0.log" Jul 24 18:11:34.283563 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.283529 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm_674513fc-7288-40ee-9a40-2747ac8f5852/pull/0.log" Jul 24 18:11:34.294587 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.294561 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm_674513fc-7288-40ee-9a40-2747ac8f5852/extract/0.log" Jul 24 18:11:34.429625 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.429540 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82_054a5c99-1bec-474b-9ae4-947e7467f0ed/util/0.log" Jul 24 18:11:34.438226 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.438201 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82_054a5c99-1bec-474b-9ae4-947e7467f0ed/pull/0.log" Jul 24 18:11:34.446124 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.446103 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82_054a5c99-1bec-474b-9ae4-947e7467f0ed/extract/0.log" Jul 24 18:11:34.569065 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.569037 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4_bc9f2fe9-d2fb-44c7-a847-1960e1be2f64/pull/0.log" Jul 24 18:11:34.577898 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.577869 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4_bc9f2fe9-d2fb-44c7-a847-1960e1be2f64/extract/0.log" Jul 24 18:11:34.585647 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.585617 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4_bc9f2fe9-d2fb-44c7-a847-1960e1be2f64/util/0.log" Jul 24 18:11:34.864319 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.864273 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-x2tdg_a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2/manager/0.log" Jul 24 18:11:34.988210 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:34.988178 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-hzpg9_22a0600b-eaa2-4f90-b2f9-bf9b02571d6b/manager/0.log" Jul 24 18:11:35.121042 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:35.120937 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jss8q_c118badb-e8bc-4a8d-95ea-ed1e95b970dc/kuadrant-console-plugin/0.log" Jul 24 18:11:35.261074 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:35.261034 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8txjb_63d947fe-7ba0-49ce-b21f-22357ee10c26/registry-server/0.log" Jul 24 18:11:35.551966 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:35.551937 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9rth9_e1d3c6b0-62fa-4107-9912-db946a0ebe59/limitador/0.log" Jul 24 18:11:35.684114 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:35.684073 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gjkz8_2e9cb16f-80db-48b7-8953-61ff7fa9189f/manager/0.log" Jul 24 18:11:36.364013 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:36.363966 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-956b69cf9-5vh8w_c4c3553b-4df1-4ccb-b888-8157824c6fa2/kube-auth-proxy/0.log" Jul 24 18:11:36.629822 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:36.629719 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rljcb_cda82824-adb8-43b7-bb20-fa840945d314/istio-proxy/1.log" Jul 24 18:11:37.050251 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:37.050210 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67844487c7-fg86z_4b6dc3ec-0cb7-4b72-bbd6-58922b496519/router/0.log" Jul 24 18:11:37.575806 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:37.575774 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-8zs67_9a398cbf-46fe-4147-89cb-0c68b116f912/main/0.log" Jul 24 18:11:44.019389 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019351 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xkcdq/must-gather-6c4bw"] Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019786 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019798 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019807 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019812 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019819 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019825 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019842 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044b66f1-5eea-4a4a-8b5a-5b7828facd3e" containerName="revoke-keys" Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019847 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="044b66f1-5eea-4a4a-8b5a-5b7828facd3e" containerName="revoke-keys" Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019858 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" Jul 24 18:11:44.019865 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019863 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" Jul 24 18:11:44.020216 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019922 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" Jul 24 18:11:44.020216 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019930 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" Jul 24 18:11:44.020216 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019937 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="044b66f1-5eea-4a4a-8b5a-5b7828facd3e" containerName="revoke-keys" Jul 24 18:11:44.020216 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.019942 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1283ebc4-b443-460d-b1b8-b8e1c3e62505" containerName="istio-proxy" Jul 24 18:11:44.020216 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.020050 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6603c3a2-5079-4bf6-bf0a-2f0863a9b51e" containerName="istio-proxy" Jul 24 18:11:44.023010 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.022992 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkcdq/must-gather-6c4bw" Jul 24 18:11:44.028882 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.028856 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xkcdq\"/\"kube-root-ca.crt\"" Jul 24 18:11:44.029061 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.029035 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xkcdq\"/\"openshift-service-ca.crt\"" Jul 24 18:11:44.029336 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.029318 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xkcdq\"/\"default-dockercfg-5k9qs\"" Jul 24 18:11:44.045134 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.045106 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkcdq/must-gather-6c4bw"] Jul 24 18:11:44.062653 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.062624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88b2ecde-90cc-43ea-be65-2f0d495db0ce-must-gather-output\") pod \"must-gather-6c4bw\" (UID: \"88b2ecde-90cc-43ea-be65-2f0d495db0ce\") " pod="openshift-must-gather-xkcdq/must-gather-6c4bw" Jul 24 18:11:44.062847 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.062658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxsp\" (UniqueName: \"kubernetes.io/projected/88b2ecde-90cc-43ea-be65-2f0d495db0ce-kube-api-access-jpxsp\") pod \"must-gather-6c4bw\" (UID: \"88b2ecde-90cc-43ea-be65-2f0d495db0ce\") " pod="openshift-must-gather-xkcdq/must-gather-6c4bw" Jul 24 18:11:44.163470 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.163440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88b2ecde-90cc-43ea-be65-2f0d495db0ce-must-gather-output\") pod \"must-gather-6c4bw\" (UID: \"88b2ecde-90cc-43ea-be65-2f0d495db0ce\") " pod="openshift-must-gather-xkcdq/must-gather-6c4bw" Jul 24 18:11:44.163470 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.163475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxsp\" (UniqueName: \"kubernetes.io/projected/88b2ecde-90cc-43ea-be65-2f0d495db0ce-kube-api-access-jpxsp\") pod \"must-gather-6c4bw\" (UID: \"88b2ecde-90cc-43ea-be65-2f0d495db0ce\") " pod="openshift-must-gather-xkcdq/must-gather-6c4bw" Jul 24 18:11:44.163873 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.163851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88b2ecde-90cc-43ea-be65-2f0d495db0ce-must-gather-output\") pod \"must-gather-6c4bw\" (UID: \"88b2ecde-90cc-43ea-be65-2f0d495db0ce\") " pod="openshift-must-gather-xkcdq/must-gather-6c4bw" Jul 24 18:11:44.172707 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.172677 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxsp\" (UniqueName: \"kubernetes.io/projected/88b2ecde-90cc-43ea-be65-2f0d495db0ce-kube-api-access-jpxsp\") pod \"must-gather-6c4bw\" (UID: \"88b2ecde-90cc-43ea-be65-2f0d495db0ce\") " pod="openshift-must-gather-xkcdq/must-gather-6c4bw" Jul 24 18:11:44.333035 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.332927 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkcdq/must-gather-6c4bw" Jul 24 18:11:44.480060 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.480025 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkcdq/must-gather-6c4bw"] Jul 24 18:11:44.481565 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:11:44.481538 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88b2ecde_90cc_43ea_be65_2f0d495db0ce.slice/crio-0c6863f1d838ab1e609624cda1c988aceddb990677643f94facbff6be52a8a93 WatchSource:0}: Error finding container 0c6863f1d838ab1e609624cda1c988aceddb990677643f94facbff6be52a8a93: Status 404 returned error can't find the container with id 0c6863f1d838ab1e609624cda1c988aceddb990677643f94facbff6be52a8a93 Jul 24 18:11:44.483863 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:44.483846 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 18:11:45.167287 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:45.167247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkcdq/must-gather-6c4bw" event={"ID":"88b2ecde-90cc-43ea-be65-2f0d495db0ce","Type":"ContainerStarted","Data":"0c6863f1d838ab1e609624cda1c988aceddb990677643f94facbff6be52a8a93"} Jul 24 18:11:46.175925 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:46.175890 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkcdq/must-gather-6c4bw" event={"ID":"88b2ecde-90cc-43ea-be65-2f0d495db0ce","Type":"ContainerStarted","Data":"3b56cba711a414d604dc78b4617da68b3f8fd69204c88d46c5add64e167be703"} Jul 24 18:11:46.176398 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:46.175942 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkcdq/must-gather-6c4bw" event={"ID":"88b2ecde-90cc-43ea-be65-2f0d495db0ce","Type":"ContainerStarted","Data":"676440c70f0e7a7bdaeceb67ed4b41f3488f2103f0067b667aa4d263aa64dd95"} Jul 24 18:11:46.200456 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:46.200398 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xkcdq/must-gather-6c4bw" podStartSLOduration=2.410027975 podStartE2EDuration="3.200380334s" totalImagesPullingTime="790.352359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:11:46.198867172 +0000 UTC m=+3262.864441831" watchObservedRunningTime="2026-07-24 18:11:46.200380334 +0000 UTC m=+3262.865954971" Jul 24 18:11:46.973877 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:46.973842 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jm92j_6a93307e-c6da-4166-861e-74ad20b421b0/global-pull-secret-syncer/0.log" Jul 24 18:11:47.177538 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:47.177504 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z5h8g_4c2e73a1-f7cf-48bc-b4f5-15bb101e0a31/konnectivity-agent/0.log" Jul 24 18:11:47.207069 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:47.207038 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-58.ec2.internal_7670db40d2e103b7e1fb032e1e25eefe/haproxy/0.log" Jul 24 18:11:51.651117 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.651079 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8_400c4845-05ff-4d9d-8a9a-a35b388ad2f6/extract/0.log" Jul 24 18:11:51.680387 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.680340 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8_400c4845-05ff-4d9d-8a9a-a35b388ad2f6/util/0.log" Jul 24 18:11:51.704446 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.704413 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zl2x8_400c4845-05ff-4d9d-8a9a-a35b388ad2f6/pull/0.log" Jul 24 18:11:51.738614 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.738528 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm_674513fc-7288-40ee-9a40-2747ac8f5852/extract/0.log" Jul 24 18:11:51.767234 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.767189 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm_674513fc-7288-40ee-9a40-2747ac8f5852/util/0.log" Jul 24 18:11:51.795993 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.795942 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07fcmm_674513fc-7288-40ee-9a40-2747ac8f5852/pull/0.log" Jul 24 18:11:51.831340 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.831315 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82_054a5c99-1bec-474b-9ae4-947e7467f0ed/extract/0.log" Jul 24 18:11:51.856950 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.856918 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82_054a5c99-1bec-474b-9ae4-947e7467f0ed/util/0.log" Jul 24 18:11:51.886245 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.886210 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73htr82_054a5c99-1bec-474b-9ae4-947e7467f0ed/pull/0.log" Jul 24 18:11:51.918229 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.918192 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4_bc9f2fe9-d2fb-44c7-a847-1960e1be2f64/extract/0.log" Jul 24 18:11:51.944697 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.944660 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4_bc9f2fe9-d2fb-44c7-a847-1960e1be2f64/util/0.log" Jul 24 18:11:51.972867 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:51.972833 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef186dv4_bc9f2fe9-d2fb-44c7-a847-1960e1be2f64/pull/0.log" Jul 24 18:11:52.367953 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:52.367920 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-x2tdg_a8f1bd49-a0f4-4af7-8d9d-d3bf949b92e2/manager/0.log" Jul 24 18:11:52.408750 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:52.408703 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-hzpg9_22a0600b-eaa2-4f90-b2f9-bf9b02571d6b/manager/0.log" Jul 24 18:11:52.444985 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:52.444953 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jss8q_c118badb-e8bc-4a8d-95ea-ed1e95b970dc/kuadrant-console-plugin/0.log" Jul 24 18:11:52.501172 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:52.501133 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8txjb_63d947fe-7ba0-49ce-b21f-22357ee10c26/registry-server/0.log" Jul 24 18:11:52.720659 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:52.720622 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9rth9_e1d3c6b0-62fa-4107-9912-db946a0ebe59/limitador/0.log" Jul 24 18:11:52.817359 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:52.817321 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gjkz8_2e9cb16f-80db-48b7-8953-61ff7fa9189f/manager/0.log" Jul 24 18:11:54.818420 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:54.818380 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-ngcgm_81f9cda0-d765-4821-b23a-62d5a9f17b12/cluster-monitoring-operator/0.log" Jul 24 18:11:54.996330 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:54.996297 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dnf9v_354d1d50-0c63-45c4-b252-620ca605b4ef/node-exporter/0.log" Jul 24 18:11:55.024393 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:55.024358 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dnf9v_354d1d50-0c63-45c4-b252-620ca605b4ef/kube-rbac-proxy/0.log" Jul 24 18:11:55.053268 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:55.053241 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dnf9v_354d1d50-0c63-45c4-b252-620ca605b4ef/init-textfile/0.log" Jul 24 18:11:55.246049 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:55.246012 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-tcxg7_7a6108b7-073e-4200-abe3-88d4f5758ffb/kube-rbac-proxy-main/0.log" Jul 24 18:11:55.279475 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:55.279443 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-tcxg7_7a6108b7-073e-4200-abe3-88d4f5758ffb/kube-rbac-proxy-self/0.log" Jul 24 18:11:55.302012 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:55.301979 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-tcxg7_7a6108b7-073e-4200-abe3-88d4f5758ffb/openshift-state-metrics/0.log" Jul 24 18:11:55.541175 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:55.541101 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-z5h9c_a8b48472-265f-4667-8191-6ff1e5b316fe/prometheus-operator/0.log" Jul 24 18:11:55.561925 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:55.561890 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-z5h9c_a8b48472-265f-4667-8191-6ff1e5b316fe/kube-rbac-proxy/0.log" Jul 24 18:11:56.991209 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:56.991150 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2"] Jul 24 18:11:56.999861 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:56.999831 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.009790 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.009632 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2"] Jul 24 18:11:57.121589 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.121551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4l4g\" (UniqueName: \"kubernetes.io/projected/459b48b5-b1da-4191-a49a-8e574d01d9c1-kube-api-access-f4l4g\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.121795 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.121612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-sys\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.121795 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.121640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-lib-modules\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.121795 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.121664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-podres\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.121795 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.121789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-proc\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.222611 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.222576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4l4g\" (UniqueName: \"kubernetes.io/projected/459b48b5-b1da-4191-a49a-8e574d01d9c1-kube-api-access-f4l4g\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.222818 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.222626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-sys\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.222818 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.222653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-lib-modules\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.222818 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.222677 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-podres\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.222995 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.222810 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-sys\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.222995 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.222819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-proc\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.222995 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.222867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-proc\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.222995 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.222886 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-lib-modules\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.222995 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.222973 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/459b48b5-b1da-4191-a49a-8e574d01d9c1-podres\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.232856 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.232823 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4l4g\" (UniqueName: \"kubernetes.io/projected/459b48b5-b1da-4191-a49a-8e574d01d9c1-kube-api-access-f4l4g\") pod \"perf-node-gather-daemonset-ps8p2\" (UID: \"459b48b5-b1da-4191-a49a-8e574d01d9c1\") " pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.315665 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.315575 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:57.518085 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.518032 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2"] Jul 24 18:11:57.522329 ip-10-0-130-58 kubenswrapper[2574]: W0724 18:11:57.522280 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod459b48b5_b1da_4191_a49a_8e574d01d9c1.slice/crio-a543ad16caefa1fe89b992bbc7bc957f645458ff8aee9a345161da7b0f989bb6 WatchSource:0}: Error finding container a543ad16caefa1fe89b992bbc7bc957f645458ff8aee9a345161da7b0f989bb6: Status 404 returned error can't find the container with id a543ad16caefa1fe89b992bbc7bc957f645458ff8aee9a345161da7b0f989bb6 Jul 24 18:11:57.761797 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.761761 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/2.log" Jul 24 18:11:57.768628 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:57.768592 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-64lhm_411738aa-268d-4f8d-b8bd-1f8cfc0ebff4/console-operator/3.log" Jul 24 18:11:58.310937 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:58.310892 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" event={"ID":"459b48b5-b1da-4191-a49a-8e574d01d9c1","Type":"ContainerStarted","Data":"d3c2bb2b626afd8672eb281f9775ef8df6f6bc52402931e4b6178fdbf81733a8"} Jul 24 18:11:58.311422 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:58.310970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" event={"ID":"459b48b5-b1da-4191-a49a-8e574d01d9c1","Type":"ContainerStarted","Data":"a543ad16caefa1fe89b992bbc7bc957f645458ff8aee9a345161da7b0f989bb6"} Jul 24 18:11:58.311849 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:58.311820 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:11:58.333785 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:58.333705 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" podStartSLOduration=2.333685279 podStartE2EDuration="2.333685279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:11:58.332864308 +0000 UTC m=+3274.998438968" watchObservedRunningTime="2026-07-24 18:11:58.333685279 +0000 UTC m=+3274.999259917" Jul 24 18:11:58.363368 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:58.363338 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b45fff8ff-86jrg_2a86c2b9-0af0-4fde-a37f-f61e949cf9c7/console/0.log" Jul 24 18:11:59.033489 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:11:59.033458 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-t67nh_e864556f-717c-46eb-81a3-eba010817bae/volume-data-source-validator/0.log" Jul 24 18:12:00.243213 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:00.243185 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbmj4_ff664f7f-5d45-4149-8c35-4d70dbc8d401/dns/0.log" Jul 24 18:12:00.268536 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:00.268502 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbmj4_ff664f7f-5d45-4149-8c35-4d70dbc8d401/kube-rbac-proxy/0.log" Jul 24 18:12:00.294022 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:00.293991 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9t9vf_6fc9ee97-5c1b-428b-ba1e-eca9eccd2e91/dns-node-resolver/0.log" Jul 24 18:12:00.933252 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:00.933208 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4n6l_4c3a820a-6416-462c-a681-f0aa4da52e8c/node-ca/0.log" Jul 24 18:12:02.205440 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:02.205412 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-956b69cf9-5vh8w_c4c3553b-4df1-4ccb-b888-8157824c6fa2/kube-auth-proxy/0.log" Jul 24 18:12:02.264955 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:02.264908 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rljcb_cda82824-adb8-43b7-bb20-fa840945d314/istio-proxy/0.log" Jul 24 18:12:02.321848 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:02.321810 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rljcb_cda82824-adb8-43b7-bb20-fa840945d314/istio-proxy/1.log" Jul 24 18:12:02.462438 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:02.462406 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67844487c7-fg86z_4b6dc3ec-0cb7-4b72-bbd6-58922b496519/router/0.log" Jul 24 18:12:03.310334 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:03.310290 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p6zz5_2f394829-084a-4947-8ac4-699f1b9708fd/serve-healthcheck-canary/0.log" Jul 24 18:12:03.820750 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:03.820691 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-hzg25_10aa3400-8f79-4e7f-97bb-d6ca24d01430/insights-operator/0.log" Jul 24 18:12:03.823455 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:03.823418 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-hzg25_10aa3400-8f79-4e7f-97bb-d6ca24d01430/insights-operator/1.log" Jul 24 18:12:04.002895 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:04.002863 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m5png_e3cd09a0-7a43-49c1-b7ba-afe6afdb9181/kube-rbac-proxy/0.log" Jul 24 18:12:04.034235 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:04.034193 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m5png_e3cd09a0-7a43-49c1-b7ba-afe6afdb9181/exporter/0.log" Jul 24 18:12:04.060071 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:04.060037 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m5png_e3cd09a0-7a43-49c1-b7ba-afe6afdb9181/extractor/0.log" Jul 24 18:12:05.347635 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:05.347593 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xkcdq/perf-node-gather-daemonset-ps8p2" Jul 24 18:12:06.242407 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:06.242373 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kbd5r_84aabd4e-2a07-48e9-a3d5-1591954a1408/manager/0.log" Jul 24 18:12:06.616593 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:06.616502 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7s2g8_5562ae78-dee0-4b9b-a693-626e0863dbf0/manager/0.log" Jul 24 18:12:07.079369 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:07.079336 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8574785b6-phgvq_6bbe55c5-07d5-4f94-a80b-15449eb8ca14/manager/0.log" Jul 24 18:12:07.100820 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:07.100791 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8574785b6-phgvq_6bbe55c5-07d5-4f94-a80b-15449eb8ca14/copy-manifests/0.log" Jul 24 18:12:08.376274 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:08.376240 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-956b69cf9-5vh8w_c4c3553b-4df1-4ccb-b888-8157824c6fa2/kube-auth-proxy/0.log" Jul 24 18:12:08.428430 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:08.428394 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rljcb_cda82824-adb8-43b7-bb20-fa840945d314/istio-proxy/0.log" Jul 24 18:12:08.484256 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:08.484217 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rljcb_cda82824-adb8-43b7-bb20-fa840945d314/istio-proxy/1.log" Jul 24 18:12:08.569204 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:08.569120 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67844487c7-fg86z_4b6dc3ec-0cb7-4b72-bbd6-58922b496519/router/0.log" Jul 24 18:12:10.228177 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:10.228150 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-956b69cf9-5vh8w_c4c3553b-4df1-4ccb-b888-8157824c6fa2/kube-auth-proxy/0.log" Jul 24 18:12:10.279206 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:10.279101 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rljcb_cda82824-adb8-43b7-bb20-fa840945d314/istio-proxy/0.log" Jul 24 18:12:10.335592 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:10.335558 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rljcb_cda82824-adb8-43b7-bb20-fa840945d314/istio-proxy/1.log" Jul 24 18:12:10.486982 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:10.486952 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67844487c7-fg86z_4b6dc3ec-0cb7-4b72-bbd6-58922b496519/router/0.log" Jul 24 18:12:14.172821 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:14.172770 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5_f6fd8a29-1f40-40e1-a279-21e27465afc4/kube-storage-version-migrator-operator/1.log" Jul 24 18:12:14.174658 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:14.174623 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-jzjb5_f6fd8a29-1f40-40e1-a279-21e27465afc4/kube-storage-version-migrator-operator/0.log" Jul 24 18:12:15.664299 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:15.664217 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mb279_5fc258ae-e5cd-4f16-84bf-cd2baf654e1a/kube-multus-additional-cni-plugins/0.log" Jul 24 18:12:15.687632 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:15.687605 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mb279_5fc258ae-e5cd-4f16-84bf-cd2baf654e1a/egress-router-binary-copy/0.log" Jul 24 18:12:15.710862 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:15.710839 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mb279_5fc258ae-e5cd-4f16-84bf-cd2baf654e1a/cni-plugins/0.log" Jul 24 18:12:15.733182 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:15.733152 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mb279_5fc258ae-e5cd-4f16-84bf-cd2baf654e1a/bond-cni-plugin/0.log" Jul 24 18:12:15.760104 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:15.760073 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mb279_5fc258ae-e5cd-4f16-84bf-cd2baf654e1a/routeoverride-cni/0.log" Jul 24 18:12:15.785367 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:15.785330 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mb279_5fc258ae-e5cd-4f16-84bf-cd2baf654e1a/whereabouts-cni-bincopy/0.log" Jul 24 18:12:15.808967 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:15.808935 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mb279_5fc258ae-e5cd-4f16-84bf-cd2baf654e1a/whereabouts-cni/0.log" Jul 24 18:12:15.862782 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:15.862746 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dxrt4_04f63ec9-fb88-475f-b6f2-89ccec28afe9/kube-multus/0.log" Jul 24 18:12:15.987260 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:15.987232 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6t9nt_bd39d23d-0f58-49d6-8d42-39caa7c31c41/network-metrics-daemon/0.log" Jul 24 18:12:16.007623 ip-10-0-130-58 kubenswrapper[2574]: I0724 18:12:16.007593 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6t9nt_bd39d23d-0f58-49d6-8d42-39caa7c31c41/kube-rbac-proxy/0.log"