Sep 11 18:15:29.873825 ip-10-0-142-253 systemd[1]: Starting Kubernetes Kubelet... Sep 11 18:15:30.299118 ip-10-0-142-253 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 18:15:30.299118 ip-10-0-142-253 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 18:15:30.299118 ip-10-0-142-253 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 18:15:30.299118 ip-10-0-142-253 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 18:15:30.299118 ip-10-0-142-253 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 18:15:30.300397 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.300283 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 18:15:30.305590 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305572 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:30.305590 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305591 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305595 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305599 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305603 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305606 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305609 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305612 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305615 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305618 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305621 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305623 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305626 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305629 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305632 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305634 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305638 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305641 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305643 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305646 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305648 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:30.305656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305651 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305659 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305662 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305665 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305668 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305671 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305673 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305676 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305679 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305682 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305684 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305687 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305689 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305692 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305694 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305697 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305700 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305704 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305707 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:30.306167 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305710 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305713 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305716 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305719 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305722 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305724 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305727 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305729 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305732 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305734 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305737 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305740 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305742 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305745 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305747 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305751 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305754 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305756 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305759 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305762 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:30.306656 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305764 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305767 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305770 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305772 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305774 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305778 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305780 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305783 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305786 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305789 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305791 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305794 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305798 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305801 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305804 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305808 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305811 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305814 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305817 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:30.307140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305819 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305822 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305824 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305827 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305830 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305833 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305835 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305837 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305841 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305845 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305847 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305850 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305852 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305855 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305858 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305861 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305865 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305868 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305871 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:30.307616 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.305874 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306623 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306631 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306634 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306637 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306640 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306643 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306645 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306648 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306651 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306654 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306658 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306661 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306663 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306666 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306669 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306672 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306675 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306677 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306680 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306683 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:30.308092 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306686 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306689 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306693 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306697 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306700 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306704 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306707 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306711 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306715 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306718 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306723 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306726 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306730 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306732 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306735 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306738 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306741 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306743 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306746 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:30.308674 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306749 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306751 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306754 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306757 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306760 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306763 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306765 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306768 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306770 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306774 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306776 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306779 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306781 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306785 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306788 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306790 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306793 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306796 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306798 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:30.309148 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306801 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306804 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306806 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306809 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306811 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306814 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306817 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306819 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306822 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306825 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306827 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306830 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306832 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306834 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306837 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306840 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306842 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306845 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306847 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306850 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:30.309718 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306855 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306858 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306860 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306863 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306866 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306868 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306871 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306874 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306876 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306880 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306882 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306885 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306888 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306890 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306893 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306895 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306898 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306901 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306903 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306906 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:30.310216 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.306908 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307302 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307307 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307310 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307313 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307316 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307319 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307321 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307324 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307327 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307330 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307332 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307337 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307339 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307342 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307345 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307347 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307351 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307353 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307356 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:30.310727 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307359 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307361 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307364 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307367 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307370 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307374 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307377 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307380 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307383 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307386 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307388 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307391 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307393 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307396 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307398 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307401 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307403 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307406 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307409 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307411 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:30.311213 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307414 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307418 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307422 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307425 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307429 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307432 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307435 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307437 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307440 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307443 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307445 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307448 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307451 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307453 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307456 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307458 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307461 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307463 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307466 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307468 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:30.311695 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307471 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307474 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307476 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307479 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307482 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307485 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307501 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307504 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307507 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307509 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307512 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307514 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307517 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307519 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307522 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307525 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307529 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307532 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307534 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307537 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:30.312199 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307540 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307544 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307547 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307550 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307553 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307556 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307558 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307561 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307564 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307567 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307569 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307572 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307574 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307577 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307580 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307582 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307584 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307587 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307590 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.307592 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:30.312746 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307659 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307696 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307703 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307708 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307712 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307715 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307720 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307724 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307731 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307735 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307738 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307742 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307745 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307748 2579 flags.go:64] FLAG: --cgroup-root="" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307751 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307754 2579 flags.go:64] FLAG: --client-ca-file="" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307758 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307761 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307765 2579 flags.go:64] FLAG: --cluster-domain="" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307768 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307771 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307774 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307778 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307781 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 18:15:30.313265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307784 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307787 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307791 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307794 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307797 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307800 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307803 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307806 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307810 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307815 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307818 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307821 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307824 2579 flags.go:64] FLAG: --enable-server="true" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307827 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307832 2579 flags.go:64] FLAG: --event-burst="100" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307835 2579 flags.go:64] FLAG: --event-qps="50" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307840 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307843 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307846 2579 flags.go:64] FLAG: --eviction-hard="" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307850 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307853 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307856 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307859 2579 flags.go:64] FLAG: --eviction-soft="" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307862 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307865 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 18:15:30.313870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307868 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307871 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307874 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307877 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307880 2579 flags.go:64] FLAG: --feature-gates="" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307884 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307887 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307890 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307893 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307896 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307900 2579 flags.go:64] FLAG: --help="false" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307903 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307906 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307909 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307912 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307915 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307920 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307922 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307925 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307928 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307931 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307934 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307938 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307942 2579 flags.go:64] FLAG: --kube-reserved="" Sep 11 18:15:30.314543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307945 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307951 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307954 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307957 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307960 2579 flags.go:64] FLAG: --lock-file="" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307963 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307966 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307969 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307974 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307977 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307980 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307983 2579 flags.go:64] FLAG: --logging-format="text" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307986 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307989 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307992 2579 flags.go:64] FLAG: --manifest-url="" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307995 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.307999 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308003 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308007 2579 flags.go:64] FLAG: --max-pods="110" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308010 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308013 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308016 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308019 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308021 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308026 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 18:15:30.315152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308029 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308037 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308040 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308043 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308046 2579 flags.go:64] FLAG: --pod-cidr="" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308049 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308055 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308059 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308063 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308067 2579 flags.go:64] FLAG: --port="10250" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308070 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308073 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-029a4676ddbc8fa04" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308076 2579 flags.go:64] FLAG: --qos-reserved="" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308079 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308082 2579 flags.go:64] FLAG: --register-node="true" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308085 2579 flags.go:64] FLAG: --register-with-taints="" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308088 2579 flags.go:64] FLAG: --registry-burst="10" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308091 2579 flags.go:64] FLAG: --registry-qps="5" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308094 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308097 2579 flags.go:64] FLAG: --reserved-memory="" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308101 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308105 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308108 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308111 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308114 2579 flags.go:64] FLAG: --runonce="false" Sep 11 18:15:30.315790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308116 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308119 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308122 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308125 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308128 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308131 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308135 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308138 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308141 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308144 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308147 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308149 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308153 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308156 2579 flags.go:64] FLAG: --system-cgroups="" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308159 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308165 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308168 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308171 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308175 2579 flags.go:64] FLAG: --tls-min-version="" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308178 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308181 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308184 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308186 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308190 2579 flags.go:64] FLAG: --v="2" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308193 2579 flags.go:64] FLAG: --version="false" Sep 11 18:15:30.316401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308197 2579 flags.go:64] FLAG: --vmodule="" Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308201 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.308204 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308339 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308343 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308346 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308348 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308351 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308354 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308357 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308360 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308362 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308365 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308369 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308371 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308374 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308376 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308379 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308381 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308384 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:30.317032 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308387 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308390 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308393 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308395 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308398 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308402 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308406 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308409 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308412 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308415 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308418 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308420 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308423 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308427 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308430 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308433 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308436 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308439 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308442 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:30.317565 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308445 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308448 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308451 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308454 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308457 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308459 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308464 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308467 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308470 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308472 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308475 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308478 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308480 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308483 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308502 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308505 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308508 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308511 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308514 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308517 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:30.318068 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308519 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308522 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308525 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308527 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308530 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308533 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308535 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308538 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308548 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308551 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308554 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308556 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308559 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308563 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308566 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308568 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308571 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308574 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308579 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:30.318633 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308581 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308584 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308587 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308589 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308592 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308595 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308597 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308600 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308603 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308605 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308608 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308610 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308613 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308616 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308619 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308621 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308624 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308626 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308629 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308632 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:30.319108 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308635 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308639 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308642 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.308645 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.309284 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.316415 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.316435 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.317277 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.317285 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317345 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317350 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317355 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317358 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317361 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:30.319612 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317364 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317367 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317370 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317373 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317375 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317378 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317381 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317383 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317386 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317389 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317391 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317394 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317397 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317399 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317402 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317404 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317407 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317410 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317412 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317415 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:30.319973 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317417 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317420 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317423 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317425 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317429 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317432 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317435 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317437 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317440 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317443 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317446 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317448 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317451 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317454 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317456 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317459 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317462 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317464 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317467 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317469 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:30.320473 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317472 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317475 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317477 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317480 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317482 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317485 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317503 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317506 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317508 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317511 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317514 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317517 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317519 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317522 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317525 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317529 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317534 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317539 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317542 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:30.320981 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317545 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317548 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317550 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317553 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317555 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317560 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317563 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317566 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317569 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317573 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317575 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317579 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317582 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317584 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317587 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317590 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317594 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317597 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317600 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:30.321481 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317603 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317606 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317609 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317612 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317615 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317617 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317620 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317623 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317625 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317628 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317631 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317634 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317637 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317640 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317642 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:30.321976 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317645 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.317650 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317768 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317773 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317776 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317779 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317820 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317823 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317827 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317830 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317832 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317835 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317838 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317840 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317843 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:30.322342 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317846 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317849 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317851 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317854 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317857 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317859 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317862 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317864 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317867 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317871 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317875 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317879 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317883 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317886 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317889 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317894 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317897 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317900 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317902 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:30.322790 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317905 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317907 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317910 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317913 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317915 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317918 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317921 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317923 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317926 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317929 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317931 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317934 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317937 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317939 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317942 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317945 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317947 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317950 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317952 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317955 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:30.323262 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317958 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317962 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317964 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317967 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317970 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317974 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317978 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317981 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317983 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317987 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317989 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317992 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317994 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.317997 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318000 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318003 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318005 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318008 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318011 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:30.323756 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318013 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318016 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318018 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318021 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318024 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318026 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318029 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318031 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318034 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318036 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318039 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318041 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318044 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318046 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318049 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318051 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318054 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318057 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318060 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:30.324217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318063 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318065 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318068 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318071 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318075 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318077 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318080 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318083 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:30.318085 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.318090 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.318209 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.320131 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.321100 2579 server.go:1019] "Starting client certificate rotation" Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.321587 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 18:15:30.324859 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.322401 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 18:15:30.325453 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.323633 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 18:15:30.327872 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.327849 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 18:15:30.336536 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.336519 2579 log.go:25] "Validated CRI v1 runtime API" Sep 11 18:15:30.339661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.339646 2579 log.go:25] "Validated CRI v1 image API" Sep 11 18:15:30.340456 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.340436 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 18:15:30.346033 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.346012 2579 fs.go:135] Filesystem UUIDs: map[28779687-8006-42a1-879b-8c3c90cb315e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cf5a07a2-c24b-48c8-b89f-35228eb34052:/dev/nvme0n1p4] Sep 11 18:15:30.346092 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.346033 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 18:15:30.351687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.351574 2579 manager.go:216] Machine: {Timestamp:2026-09-11 18:15:30.350302823 +0000 UTC m=+0.348193319 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099821 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec203ea28cca698b0d239efd7b37a6b4 SystemUUID:ec203ea2-8cca-698b-0d23-9efd7b37a6b4 BootID:fe5256a4-b022-4899-b8f6-e98503a4763d Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:09:1a:3b:dd:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:09:1a:3b:dd:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:23:64:77:cf:8d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 18:15:30.351687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.351684 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 18:15:30.351801 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.351773 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 18:15:30.352667 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.352651 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 18:15:30.354744 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.354712 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 18:15:30.354904 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.354745 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 18:15:30.356111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.356101 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 18:15:30.356145 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.356115 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 18:15:30.356145 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.356122 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 18:15:30.356881 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.356867 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 18:15:30.357689 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.357679 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 18:15:30.362392 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.362380 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 18:15:30.362517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.362507 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 18:15:30.362557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.362535 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 11 18:15:30.362557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.362552 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 18:15:30.362629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.362589 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 18:15:30.362629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.362605 2579 kubelet.go:398] "Adding apiserver pod source" Sep 11 18:15:30.362629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.362620 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 18:15:30.363060 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.363046 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 18:15:30.363155 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.363065 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 18:15:30.366203 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.366185 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 18:15:30.368719 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.368706 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 18:15:30.368762 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.368736 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 18:15:30.370711 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370690 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 18:15:30.370711 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370715 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370725 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370733 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370745 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370753 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370773 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370781 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370813 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370823 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370836 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 18:15:30.370858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.370846 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 18:15:30.371826 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.371811 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 18:15:30.371929 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.371919 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 18:15:30.375144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.375118 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-253.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 18:15:30.375662 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.375640 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 18:15:30.375662 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.375649 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 18:15:30.376883 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.376866 2579 server.go:1268] "Started kubelet" Sep 11 18:15:30.377006 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.376948 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 18:15:30.377111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.377019 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 18:15:30.377111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.377089 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 18:15:30.377870 ip-10-0-142-253 systemd[1]: Started Kubernetes Kubelet. Sep 11 18:15:30.379001 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.378981 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 18:15:30.379636 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.379622 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 11 18:15:30.383377 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.383358 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 18:15:30.383927 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.382920 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-253.ec2.internal.18d456ed94d043e9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-253.ec2.internal,UID:ip-10-0-142-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-253.ec2.internal,},FirstTimestamp:2026-09-11 18:15:30.376844265 +0000 UTC m=+0.374734737,LastTimestamp:2026-09-11 18:15:30.376844265 +0000 UTC m=+0.374734737,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-253.ec2.internal,}" Sep 11 18:15:30.384591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.384563 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 18:15:30.384591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.384580 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 18:15:30.385146 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.384990 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:30.385146 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.385100 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 18:15:30.385146 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.385145 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 18:15:30.386170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.385162 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 18:15:30.386170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.385485 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 11 18:15:30.386170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.385509 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 11 18:15:30.386453 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.386438 2579 factory.go:55] Registering systemd factory Sep 11 18:15:30.386514 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.386457 2579 factory.go:223] Registration of the systemd container factory successfully Sep 11 18:15:30.386705 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.386688 2579 factory.go:153] Registering CRI-O factory Sep 11 18:15:30.386705 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.386706 2579 factory.go:223] Registration of the crio container factory successfully Sep 11 18:15:30.386835 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.386757 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 18:15:30.386835 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.386775 2579 factory.go:103] Registering Raw factory Sep 11 18:15:30.386835 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.386791 2579 manager.go:1196] Started watching for new ooms in manager Sep 11 18:15:30.387225 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.387213 2579 manager.go:349] Starting recovery of all containers Sep 11 18:15:30.389345 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.389312 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 18:15:30.389345 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.389333 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 18:15:30.396671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.396652 2579 manager.go:354] Recovery completed Sep 11 18:15:30.404052 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.404033 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:30.406989 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.406969 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:30.407077 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.407004 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:30.407077 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.407020 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:30.407783 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.407765 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 18:15:30.407783 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.407782 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 18:15:30.407911 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.407802 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 18:15:30.407911 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.407774 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:30.408621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.408604 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:30.408697 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.408637 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:30.408697 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.408653 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:30.409217 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.409128 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-253.ec2.internal.18d456ed969c3874 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-253.ec2.internal,UID:ip-10-0-142-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-253.ec2.internal,},FirstTimestamp:2026-09-11 18:15:30.406987892 +0000 UTC m=+0.404878365,LastTimestamp:2026-09-11 18:15:30.406987892 +0000 UTC m=+0.404878365,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-253.ec2.internal,}" Sep 11 18:15:30.410026 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.410013 2579 policy_none.go:49] "None policy: Start" Sep 11 18:15:30.410067 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.410032 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 18:15:30.410067 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.410052 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 18:15:30.411995 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.411982 2579 policy_none.go:47] "Start" Sep 11 18:15:30.417817 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.417732 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-253.ec2.internal.18d456ed969c944f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-253.ec2.internal,UID:ip-10-0-142-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-253.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-253.ec2.internal,},FirstTimestamp:2026-09-11 18:15:30.407011407 +0000 UTC m=+0.404901879,LastTimestamp:2026-09-11 18:15:30.407011407 +0000 UTC m=+0.404901879,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-253.ec2.internal,}" Sep 11 18:15:30.419399 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.419378 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mlbtb" Sep 11 18:15:30.420546 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.420433 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-253.ec2.internal.18d456ed969ccbbb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-253.ec2.internal,UID:ip-10-0-142-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-253.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-253.ec2.internal,},FirstTimestamp:2026-09-11 18:15:30.407025595 +0000 UTC m=+0.404916068,LastTimestamp:2026-09-11 18:15:30.407025595 +0000 UTC m=+0.404916068,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-253.ec2.internal,}" Sep 11 18:15:30.427369 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.427288 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-253.ec2.internal.18d456ed969c3874\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-253.ec2.internal.18d456ed969c3874 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-253.ec2.internal,UID:ip-10-0-142-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-253.ec2.internal,},FirstTimestamp:2026-09-11 18:15:30.406987892 +0000 UTC m=+0.404878365,LastTimestamp:2026-09-11 18:15:30.408620994 +0000 UTC m=+0.406511468,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-253.ec2.internal,}" Sep 11 18:15:30.428097 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.428077 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mlbtb" Sep 11 18:15:30.431712 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.431634 2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-253.ec2.internal.18d456ed969c944f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-253.ec2.internal.18d456ed969c944f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-253.ec2.internal,UID:ip-10-0-142-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-253.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-253.ec2.internal,},FirstTimestamp:2026-09-11 18:15:30.407011407 +0000 UTC m=+0.404901879,LastTimestamp:2026-09-11 18:15:30.408644273 +0000 UTC m=+0.406534746,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-253.ec2.internal,}" Sep 11 18:15:30.435170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.435156 2579 manager.go:337] "Starting Device Plugin manager" Sep 11 18:15:30.435252 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.435208 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 18:15:30.435252 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.435221 2579 server.go:87] "Starting device plugin registration server" Sep 11 18:15:30.435618 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.435602 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 18:15:30.435691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.435617 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 18:15:30.435922 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.435736 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 18:15:30.435922 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.435814 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 18:15:30.435922 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.435823 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 18:15:30.437712 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.437675 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 18:15:30.464247 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.437728 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:30.464247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.449670 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 18:15:30.464247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.450757 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 18:15:30.464247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.450796 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 18:15:30.464247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.450819 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 18:15:30.464247 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.450858 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 18:15:30.464247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.453164 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:30.536563 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.536516 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:30.537586 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.537567 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:30.537722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.537597 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:30.537722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.537609 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:30.537722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.537642 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.546442 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.546423 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.546566 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.546450 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-253.ec2.internal\": node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:30.551599 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.551534 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal"] Sep 11 18:15:30.551661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.551614 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:30.553248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.553232 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:30.553317 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.553261 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:30.553317 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.553276 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:30.555588 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.555574 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:30.555712 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.555696 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.555761 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.555730 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:30.556267 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.556250 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:30.556342 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.556278 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:30.556342 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.556288 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:30.556342 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.556308 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:30.556342 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.556335 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:30.556531 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.556355 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:30.558500 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.558478 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.558565 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.558512 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:30.559229 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.559213 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:30.559287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.559235 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:30.559287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.559250 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:30.566752 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.566736 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:30.585291 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.585268 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-253.ec2.internal\" not found" node="ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.587266 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.587249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d99a6b4b5bc4fea990e0ac691bcb542-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal\" (UID: \"5d99a6b4b5bc4fea990e0ac691bcb542\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.587336 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.587277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d99a6b4b5bc4fea990e0ac691bcb542-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal\" (UID: \"5d99a6b4b5bc4fea990e0ac691bcb542\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.587336 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.587301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b275f54185b5d5dddefd04d62d766a8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-253.ec2.internal\" (UID: \"6b275f54185b5d5dddefd04d62d766a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.589818 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.589803 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-253.ec2.internal\" not found" node="ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.667625 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.667596 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:30.688302 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.688278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d99a6b4b5bc4fea990e0ac691bcb542-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal\" (UID: \"5d99a6b4b5bc4fea990e0ac691bcb542\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.688398 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.688305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d99a6b4b5bc4fea990e0ac691bcb542-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal\" (UID: \"5d99a6b4b5bc4fea990e0ac691bcb542\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.688398 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.688322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b275f54185b5d5dddefd04d62d766a8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-253.ec2.internal\" (UID: \"6b275f54185b5d5dddefd04d62d766a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.688398 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.688386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b275f54185b5d5dddefd04d62d766a8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-253.ec2.internal\" (UID: \"6b275f54185b5d5dddefd04d62d766a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.688511 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.688396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d99a6b4b5bc4fea990e0ac691bcb542-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal\" (UID: \"5d99a6b4b5bc4fea990e0ac691bcb542\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.688511 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.688386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d99a6b4b5bc4fea990e0ac691bcb542-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal\" (UID: \"5d99a6b4b5bc4fea990e0ac691bcb542\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.768480 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.768434 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:30.869306 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.869214 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:30.888557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.888529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.892137 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:30.892108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal" Sep 11 18:15:30.969790 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:30.969740 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:32.841536 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:32.841503 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:32.942049 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:32.941976 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:33.042629 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:33.042591 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:33.092061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.092040 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 18:15:33.092420 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.092177 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 18:15:33.143594 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:33.143564 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-253.ec2.internal\" not found" Sep 11 18:15:33.197173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.197113 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:33.203502 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.203343 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:10:32 +0000 UTC" deadline="2028-02-06 13:43:04.45162826 +0000 UTC" Sep 11 18:15:33.203502 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.203502 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12307h27m31.24813041s" Sep 11 18:15:33.212828 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:33.212798 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d99a6b4b5bc4fea990e0ac691bcb542.slice/crio-0bcafcd2bf0b7ac2895759b8a81e7fdf476f4b39be0d68d47d03ae82c62bd4c7 WatchSource:0}: Error finding container 0bcafcd2bf0b7ac2895759b8a81e7fdf476f4b39be0d68d47d03ae82c62bd4c7: Status 404 returned error can't find the container with id 0bcafcd2bf0b7ac2895759b8a81e7fdf476f4b39be0d68d47d03ae82c62bd4c7 Sep 11 18:15:33.213070 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:33.213054 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b275f54185b5d5dddefd04d62d766a8.slice/crio-b8870234d96c65dc0fc954751aea885151d71636cf89b5665ff9f09ac34f5ce4 WatchSource:0}: Error finding container b8870234d96c65dc0fc954751aea885151d71636cf89b5665ff9f09ac34f5ce4: Status 404 returned error can't find the container with id b8870234d96c65dc0fc954751aea885151d71636cf89b5665ff9f09ac34f5ce4 Sep 11 18:15:33.217608 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.217593 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:15:33.225061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.225018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" event={"ID":"5d99a6b4b5bc4fea990e0ac691bcb542","Type":"ContainerStarted","Data":"0bcafcd2bf0b7ac2895759b8a81e7fdf476f4b39be0d68d47d03ae82c62bd4c7"} Sep 11 18:15:33.225939 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.225923 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal" event={"ID":"6b275f54185b5d5dddefd04d62d766a8","Type":"ContainerStarted","Data":"b8870234d96c65dc0fc954751aea885151d71636cf89b5665ff9f09ac34f5ce4"} Sep 11 18:15:33.256531 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.256511 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" Sep 11 18:15:33.272025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.272007 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:33.272182 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.272170 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 18:15:33.274403 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.274390 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal" Sep 11 18:15:33.284771 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.284759 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 18:15:33.559963 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:33.559887 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:34.134887 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.134848 2579 apiserver.go:52] "Watching apiserver" Sep 11 18:15:34.143165 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.143141 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 18:15:34.143542 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.143510 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-bw29p","kube-system/global-pull-secret-syncer-jphcv","kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv","openshift-network-diagnostics/network-check-target-rn9px","openshift-network-operator/iptables-alerter-l6ff2","openshift-ovn-kubernetes/ovnkube-node-d4sfd","kube-system/konnectivity-agent-cgv47","openshift-cluster-node-tuning-operator/tuned-bn5pf","openshift-dns/node-resolver-mn77k","openshift-image-registry/node-ca-wkmbt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal","openshift-multus/multus-additional-cni-plugins-kdttd","openshift-multus/multus-kj7zc"] Sep 11 18:15:34.146125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.146101 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.148366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.148348 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.150521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.150497 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.152465 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.152446 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 18:15:34.152913 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.152895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.153491 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.153325 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 18:15:34.153491 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.153344 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 18:15:34.153962 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.153941 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 18:15:34.154082 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.154014 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2s75l\"" Sep 11 18:15:34.154170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.154149 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 18:15:34.154170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.154170 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 18:15:34.154310 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.154149 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 18:15:34.154310 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.154196 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 18:15:34.154310 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.154269 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 18:15:34.154962 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.154936 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 18:15:34.155035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.154944 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 18:15:34.155035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.154967 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 18:15:34.155406 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.155389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:34.155487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.155458 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4f2fg\"" Sep 11 18:15:34.155558 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.155538 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gkf6t\"" Sep 11 18:15:34.155630 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.155455 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:34.157011 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.156985 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 18:15:34.157785 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.157762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.161405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.159371 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 18:15:34.161405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.159907 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 18:15:34.161405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.159952 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lpfpx\"" Sep 11 18:15:34.161405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.159996 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 18:15:34.161405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.160462 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 18:15:34.161405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.160928 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2vckq\"" Sep 11 18:15:34.161405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.161153 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 18:15:34.161405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.161377 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:15:34.161942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.161806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.164975 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.164954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:34.165064 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.165039 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:34.166366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.166348 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 18:15:34.166366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.166362 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:15:34.166500 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.166348 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gs4jh\"" Sep 11 18:15:34.167424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.167407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.167527 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.167472 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:34.168581 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-env-overrides\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.168664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wcs4\" (UniqueName: \"kubernetes.io/projected/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-kube-api-access-8wcs4\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.168664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-run-k8s-cni-cncf-io\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.168664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-daemon-config\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.168889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-var-lib-openvswitch\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.168889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-run-ovn\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.168889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6b7d068-b3ff-4340-86b4-6919733efb4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.168889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-socket-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.168889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8fa86ed-523c-4c1b-a59f-3d08c1039f56-iptables-alerter-script\") pod \"iptables-alerter-l6ff2\" (UID: \"f8fa86ed-523c-4c1b-a59f-3d08c1039f56\") " pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.168889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-run-systemd\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.168889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-etc-openvswitch\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-var-lib-kubelet\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.169147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.168965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjbjl\" (UniqueName: \"kubernetes.io/projected/a245d879-6ccd-4044-b460-19db9dfdc44d-kube-api-access-hjbjl\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.169147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-cni-netd\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-ovnkube-script-lib\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-kubelet\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-systemd-units\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-registration-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.169147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-cni-bin\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-socket-dir-parent\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-run-multus-certs\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-ovnkube-config\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6b7d068-b3ff-4340-86b4-6919733efb4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtr66\" (UniqueName: \"kubernetes.io/projected/e6b7d068-b3ff-4340-86b4-6919733efb4a-kube-api-access-xtr66\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-sys-fs\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdb19ff5-33de-41c7-aebf-0756d014e6f3-cni-binary-copy\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-etc-kubernetes\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-run-netns\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-conf-dir\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-device-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-slash\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6b7d068-b3ff-4340-86b4-6919733efb4a-cnibin\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e6b7d068-b3ff-4340-86b4-6919733efb4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmbp\" (UniqueName: \"kubernetes.io/projected/cdb19ff5-33de-41c7-aebf-0756d014e6f3-kube-api-access-pkmbp\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.169598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-etc-selinux\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-system-cni-dir\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-node-log\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6b7d068-b3ff-4340-86b4-6919733efb4a-system-cni-dir\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-run-netns\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-var-lib-cni-multus\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-cnibin\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-ovn-node-metrics-cert\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e6b7d068-b3ff-4340-86b4-6919733efb4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-hostroot\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pgc\" (UniqueName: \"kubernetes.io/projected/f8fa86ed-523c-4c1b-a59f-3d08c1039f56-kube-api-access-n7pgc\") pod \"iptables-alerter-l6ff2\" (UID: \"f8fa86ed-523c-4c1b-a59f-3d08c1039f56\") " pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-log-socket\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.169869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-cni-dir\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.170104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:34.170296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.170128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8fa86ed-523c-4c1b-a59f-3d08c1039f56-host-slash\") pod \"iptables-alerter-l6ff2\" (UID: \"f8fa86ed-523c-4c1b-a59f-3d08c1039f56\") " pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.170848 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.170149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-run-openvswitch\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.170848 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.170172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-var-lib-cni-bin\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.172140 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.172122 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.174388 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.174362 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:34.175691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.175666 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 18:15:34.175691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.175686 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 18:15:34.175874 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.175739 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d4pvq\"" Sep 11 18:15:34.175874 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.175670 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 18:15:34.176010 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.175932 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h78k6\"" Sep 11 18:15:34.176215 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.176200 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 18:15:34.176606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.176586 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 18:15:34.176775 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.176623 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 18:15:34.176851 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.176842 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g5lx4\"" Sep 11 18:15:34.178213 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.178195 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 18:15:34.185020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.184922 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 18:15:34.205560 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.205539 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-888l7" Sep 11 18:15:34.219302 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.219279 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-888l7" Sep 11 18:15:34.257828 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.257805 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 18:15:34.270553 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8fa86ed-523c-4c1b-a59f-3d08c1039f56-iptables-alerter-script\") pod \"iptables-alerter-l6ff2\" (UID: \"f8fa86ed-523c-4c1b-a59f-3d08c1039f56\") " pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.270746 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-run-systemd\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.270746 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-etc-openvswitch\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.270891 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.270891 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-var-lib-kubelet\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.270891 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-sysconfig\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.270891 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.271061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbjl\" (UniqueName: \"kubernetes.io/projected/a245d879-6ccd-4044-b460-19db9dfdc44d-kube-api-access-hjbjl\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.271061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-cni-netd\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-ovnkube-script-lib\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.270974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-sysctl-d\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.271061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxh8\" (UniqueName: \"kubernetes.io/projected/27a6d135-2518-45ee-b801-709400c2a1a0-kube-api-access-qkxh8\") pod \"node-resolver-mn77k\" (UID: \"27a6d135-2518-45ee-b801-709400c2a1a0\") " pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.271061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-kubelet\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-systemd-units\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-registration-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.271357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8fa86ed-523c-4c1b-a59f-3d08c1039f56-iptables-alerter-script\") pod \"iptables-alerter-l6ff2\" (UID: \"f8fa86ed-523c-4c1b-a59f-3d08c1039f56\") " pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.271357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-etc-openvswitch\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271200 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-cni-bin\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-run-systemd\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-cni-bin\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-var-lib-kubelet\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.271357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271328 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-cni-netd\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-kubelet\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271447 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-registration-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-systemd-units\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-socket-dir-parent\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-run-multus-certs\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-modprobe-d\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-sys\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271656 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-tuned\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz4fq\" (UniqueName: \"kubernetes.io/projected/03fbffc5-9471-44da-b5b0-4b881b17898c-kube-api-access-lz4fq\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-ovnkube-config\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.271760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6b7d068-b3ff-4340-86b4-6919733efb4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtr66\" (UniqueName: \"kubernetes.io/projected/e6b7d068-b3ff-4340-86b4-6919733efb4a-kube-api-access-xtr66\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7mz\" (UniqueName: \"kubernetes.io/projected/ae2f026a-4049-47cb-a43b-7eb14472f1ef-kube-api-access-2j7mz\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27a6d135-2518-45ee-b801-709400c2a1a0-hosts-file\") pod \"node-resolver-mn77k\" (UID: \"27a6d135-2518-45ee-b801-709400c2a1a0\") " pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-ovnkube-script-lib\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271942 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34e112ec-e721-4ded-a94d-f259b4e2e2b8-agent-certs\") pod \"konnectivity-agent-cgv47\" (UID: \"34e112ec-e721-4ded-a94d-f259b4e2e2b8\") " pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.271992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-sys-fs\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdb19ff5-33de-41c7-aebf-0756d014e6f3-cni-binary-copy\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-etc-kubernetes\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-lib-modules\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-kubelet-config\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272142 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34e112ec-e721-4ded-a94d-f259b4e2e2b8-konnectivity-ca\") pod \"konnectivity-agent-cgv47\" (UID: \"34e112ec-e721-4ded-a94d-f259b4e2e2b8\") " pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:34.272228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272205 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-sys-fs\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.272780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-socket-dir-parent\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.272780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-run-multus-certs\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.272780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdb19ff5-33de-41c7-aebf-0756d014e6f3-cni-binary-copy\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.272926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-etc-kubernetes\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.272926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-run-netns\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.272926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-conf-dir\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.272926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a6d135-2518-45ee-b801-709400c2a1a0-tmp-dir\") pod \"node-resolver-mn77k\" (UID: \"27a6d135-2518-45ee-b801-709400c2a1a0\") " pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.273099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-device-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.273099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.272986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-slash\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.273099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6b7d068-b3ff-4340-86b4-6919733efb4a-cnibin\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.273099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e6b7d068-b3ff-4340-86b4-6919733efb4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.273099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-run-netns\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.273297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-device-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.273297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmbp\" (UniqueName: \"kubernetes.io/projected/cdb19ff5-33de-41c7-aebf-0756d014e6f3-kube-api-access-pkmbp\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.273297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273154 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-slash\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.273297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:34.273297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273039 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-ovnkube-config\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.273297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-etc-selinux\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.273297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-conf-dir\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.273644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6b7d068-b3ff-4340-86b4-6919733efb4a-cnibin\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.273644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-system-cni-dir\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.273644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-node-log\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.273644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273525 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-system-cni-dir\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.273644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.273644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-node-log\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.273644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273566 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-etc-selinux\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.273644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6b7d068-b3ff-4340-86b4-6919733efb4a-system-cni-dir\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.273644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273612 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6b7d068-b3ff-4340-86b4-6919733efb4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.274080 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6b7d068-b3ff-4340-86b4-6919733efb4a-system-cni-dir\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.274080 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-run-netns\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.274080 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-var-lib-cni-multus\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.274080 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273793 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-var-lib-cni-multus\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.274576 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.273628 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.274576 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-systemd\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.274728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-run-netns\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.274728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-cnibin\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.274728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274638 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-cnibin\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.274728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.274728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-ovn-node-metrics-cert\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.274728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e6b7d068-b3ff-4340-86b4-6919733efb4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274764 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e6b7d068-b3ff-4340-86b4-6919733efb4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-hostroot\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-kubernetes\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-run\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-host\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-hostroot\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbaaf296-872b-4b25-b8da-17a8f77ad1d3-host\") pod \"node-ca-wkmbt\" (UID: \"cbaaf296-872b-4b25-b8da-17a8f77ad1d3\") " pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pgc\" (UniqueName: \"kubernetes.io/projected/f8fa86ed-523c-4c1b-a59f-3d08c1039f56-kube-api-access-n7pgc\") pod \"iptables-alerter-l6ff2\" (UID: \"f8fa86ed-523c-4c1b-a59f-3d08c1039f56\") " pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.274995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-log-socket\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.275025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-cni-dir\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8fa86ed-523c-4c1b-a59f-3d08c1039f56-host-slash\") pod \"iptables-alerter-l6ff2\" (UID: \"f8fa86ed-523c-4c1b-a59f-3d08c1039f56\") " pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-run-openvswitch\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-var-lib-cni-bin\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-dbus\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbaaf296-872b-4b25-b8da-17a8f77ad1d3-serviceca\") pod \"node-ca-wkmbt\" (UID: \"cbaaf296-872b-4b25-b8da-17a8f77ad1d3\") " pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-env-overrides\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wcs4\" (UniqueName: \"kubernetes.io/projected/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-kube-api-access-8wcs4\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-run-k8s-cni-cncf-io\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-daemon-config\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8fa86ed-523c-4c1b-a59f-3d08c1039f56-host-slash\") pod \"iptables-alerter-l6ff2\" (UID: \"f8fa86ed-523c-4c1b-a59f-3d08c1039f56\") " pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e6b7d068-b3ff-4340-86b4-6919733efb4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275390 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-log-socket\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275397 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-cni-dir\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-var-lib-cni-bin\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-run-openvswitch\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.275521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-var-lib-kubelet\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxf5\" (UniqueName: \"kubernetes.io/projected/cbaaf296-872b-4b25-b8da-17a8f77ad1d3-kube-api-access-vbxf5\") pod \"node-ca-wkmbt\" (UID: \"cbaaf296-872b-4b25-b8da-17a8f77ad1d3\") " pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275621 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-env-overrides\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275657 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdb19ff5-33de-41c7-aebf-0756d014e6f3-host-run-k8s-cni-cncf-io\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-var-lib-openvswitch\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275665 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-run-ovn\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275730 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-var-lib-openvswitch\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6b7d068-b3ff-4340-86b4-6919733efb4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275761 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-run-ovn\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-sysctl-conf\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275824 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdb19ff5-33de-41c7-aebf-0756d014e6f3-multus-daemon-config\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae2f026a-4049-47cb-a43b-7eb14472f1ef-tmp\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.275937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-socket-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.276073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a245d879-6ccd-4044-b460-19db9dfdc44d-socket-dir\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.276340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.276237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6b7d068-b3ff-4340-86b4-6919733efb4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.279366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.279343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-ovn-node-metrics-cert\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.283294 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.283257 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:34.283294 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.283286 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:34.283294 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.283300 2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5nml for pod openshift-network-diagnostics/network-check-target-rn9px: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:34.283639 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.283376 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml podName:32321d1e-f561-4a92-85a2-19420d4d2220 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:34.783345052 +0000 UTC m=+3.010021414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q5nml" (UniqueName: "kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml") pod "network-check-target-rn9px" (UID: "32321d1e-f561-4a92-85a2-19420d4d2220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:34.284214 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.284175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtr66\" (UniqueName: \"kubernetes.io/projected/e6b7d068-b3ff-4340-86b4-6919733efb4a-kube-api-access-xtr66\") pod \"multus-additional-cni-plugins-kdttd\" (UID: \"e6b7d068-b3ff-4340-86b4-6919733efb4a\") " pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.284517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.284484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbjl\" (UniqueName: \"kubernetes.io/projected/a245d879-6ccd-4044-b460-19db9dfdc44d-kube-api-access-hjbjl\") pod \"aws-ebs-csi-driver-node-gsgsv\" (UID: \"a245d879-6ccd-4044-b460-19db9dfdc44d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.284774 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.284757 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pgc\" (UniqueName: \"kubernetes.io/projected/f8fa86ed-523c-4c1b-a59f-3d08c1039f56-kube-api-access-n7pgc\") pod \"iptables-alerter-l6ff2\" (UID: \"f8fa86ed-523c-4c1b-a59f-3d08c1039f56\") " pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.285013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.284987 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmbp\" (UniqueName: \"kubernetes.io/projected/cdb19ff5-33de-41c7-aebf-0756d014e6f3-kube-api-access-pkmbp\") pod \"multus-kj7zc\" (UID: \"cdb19ff5-33de-41c7-aebf-0756d014e6f3\") " pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.286439 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.286422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wcs4\" (UniqueName: \"kubernetes.io/projected/377ee3bf-f751-4c9c-9552-eb50fa46d6d5-kube-api-access-8wcs4\") pod \"ovnkube-node-d4sfd\" (UID: \"377ee3bf-f751-4c9c-9552-eb50fa46d6d5\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.376201 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-kubernetes\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376201 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-run\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-host\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbaaf296-872b-4b25-b8da-17a8f77ad1d3-host\") pod \"node-ca-wkmbt\" (UID: \"cbaaf296-872b-4b25-b8da-17a8f77ad1d3\") " pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-host\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-kubernetes\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-dbus\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376355 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-run\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbaaf296-872b-4b25-b8da-17a8f77ad1d3-serviceca\") pod \"node-ca-wkmbt\" (UID: \"cbaaf296-872b-4b25-b8da-17a8f77ad1d3\") " pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbaaf296-872b-4b25-b8da-17a8f77ad1d3-host\") pod \"node-ca-wkmbt\" (UID: \"cbaaf296-872b-4b25-b8da-17a8f77ad1d3\") " pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-var-lib-kubelet\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxf5\" (UniqueName: \"kubernetes.io/projected/cbaaf296-872b-4b25-b8da-17a8f77ad1d3-kube-api-access-vbxf5\") pod \"node-ca-wkmbt\" (UID: \"cbaaf296-872b-4b25-b8da-17a8f77ad1d3\") " pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-sysctl-conf\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-dbus\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae2f026a-4049-47cb-a43b-7eb14472f1ef-tmp\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-sysconfig\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-var-lib-kubelet\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-sysctl-d\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376603 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-sysconfig\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxh8\" (UniqueName: \"kubernetes.io/projected/27a6d135-2518-45ee-b801-709400c2a1a0-kube-api-access-qkxh8\") pod \"node-resolver-mn77k\" (UID: \"27a6d135-2518-45ee-b801-709400c2a1a0\") " pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-modprobe-d\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-sys\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.376669 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-tuned\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376695 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-sysctl-conf\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.376755 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret podName:f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:34.876734429 +0000 UTC m=+3.103410771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret") pod "global-pull-secret-syncer-jphcv" (UID: "f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-modprobe-d\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.376997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz4fq\" (UniqueName: \"kubernetes.io/projected/03fbffc5-9471-44da-b5b0-4b881b17898c-kube-api-access-lz4fq\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-sys\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7mz\" (UniqueName: \"kubernetes.io/projected/ae2f026a-4049-47cb-a43b-7eb14472f1ef-kube-api-access-2j7mz\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbaaf296-872b-4b25-b8da-17a8f77ad1d3-serviceca\") pod \"node-ca-wkmbt\" (UID: \"cbaaf296-872b-4b25-b8da-17a8f77ad1d3\") " pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376891 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-sysctl-d\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27a6d135-2518-45ee-b801-709400c2a1a0-hosts-file\") pod \"node-resolver-mn77k\" (UID: \"27a6d135-2518-45ee-b801-709400c2a1a0\") " pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34e112ec-e721-4ded-a94d-f259b4e2e2b8-agent-certs\") pod \"konnectivity-agent-cgv47\" (UID: \"34e112ec-e721-4ded-a94d-f259b4e2e2b8\") " pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-lib-modules\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376974 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27a6d135-2518-45ee-b801-709400c2a1a0-hosts-file\") pod \"node-resolver-mn77k\" (UID: \"27a6d135-2518-45ee-b801-709400c2a1a0\") " pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.376981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-kubelet-config\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.377016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34e112ec-e721-4ded-a94d-f259b4e2e2b8-konnectivity-ca\") pod \"konnectivity-agent-cgv47\" (UID: \"34e112ec-e721-4ded-a94d-f259b4e2e2b8\") " pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.377036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-kubelet-config\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.377046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a6d135-2518-45ee-b801-709400c2a1a0-tmp-dir\") pod \"node-resolver-mn77k\" (UID: \"27a6d135-2518-45ee-b801-709400c2a1a0\") " pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.377076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.377084 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-lib-modules\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.377115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-systemd\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.377186 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:34.377812 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.377244 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs podName:03fbffc5-9471-44da-b5b0-4b881b17898c nodeName:}" failed. No retries permitted until 2026-09-11 18:15:34.87722186 +0000 UTC m=+3.103898210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs") pod "network-metrics-daemon-bw29p" (UID: "03fbffc5-9471-44da-b5b0-4b881b17898c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:34.378612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.377227 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-systemd\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.378612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.377394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a6d135-2518-45ee-b801-709400c2a1a0-tmp-dir\") pod \"node-resolver-mn77k\" (UID: \"27a6d135-2518-45ee-b801-709400c2a1a0\") " pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.378612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.377556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34e112ec-e721-4ded-a94d-f259b4e2e2b8-konnectivity-ca\") pod \"konnectivity-agent-cgv47\" (UID: \"34e112ec-e721-4ded-a94d-f259b4e2e2b8\") " pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:34.379325 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.379300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae2f026a-4049-47cb-a43b-7eb14472f1ef-tmp\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.379483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.379466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae2f026a-4049-47cb-a43b-7eb14472f1ef-etc-tuned\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.379546 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.379474 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34e112ec-e721-4ded-a94d-f259b4e2e2b8-agent-certs\") pod \"konnectivity-agent-cgv47\" (UID: \"34e112ec-e721-4ded-a94d-f259b4e2e2b8\") " pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:34.388513 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.388458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxh8\" (UniqueName: \"kubernetes.io/projected/27a6d135-2518-45ee-b801-709400c2a1a0-kube-api-access-qkxh8\") pod \"node-resolver-mn77k\" (UID: \"27a6d135-2518-45ee-b801-709400c2a1a0\") " pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.389134 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.389111 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7mz\" (UniqueName: \"kubernetes.io/projected/ae2f026a-4049-47cb-a43b-7eb14472f1ef-kube-api-access-2j7mz\") pod \"tuned-bn5pf\" (UID: \"ae2f026a-4049-47cb-a43b-7eb14472f1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.389250 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.389135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxf5\" (UniqueName: \"kubernetes.io/projected/cbaaf296-872b-4b25-b8da-17a8f77ad1d3-kube-api-access-vbxf5\") pod \"node-ca-wkmbt\" (UID: \"cbaaf296-872b-4b25-b8da-17a8f77ad1d3\") " pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.389446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.389429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz4fq\" (UniqueName: \"kubernetes.io/projected/03fbffc5-9471-44da-b5b0-4b881b17898c-kube-api-access-lz4fq\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:34.400471 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.400449 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:34.457720 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.457682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:34.464490 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.464469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kdttd" Sep 11 18:15:34.472613 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.472595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kj7zc" Sep 11 18:15:34.478640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.478624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" Sep 11 18:15:34.485963 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.485945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l6ff2" Sep 11 18:15:34.492695 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.492677 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" Sep 11 18:15:34.498525 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.498504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mn77k" Sep 11 18:15:34.505118 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.505100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkmbt" Sep 11 18:15:34.509977 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.509961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:34.880290 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.880214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:34.880290 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.880256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:34.880534 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:34.880306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:34.880534 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.880386 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:34.880534 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.880415 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:34.880534 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.880425 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:34.880534 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.880458 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs podName:03fbffc5-9471-44da-b5b0-4b881b17898c nodeName:}" failed. No retries permitted until 2026-09-11 18:15:35.880445244 +0000 UTC m=+4.107121562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs") pod "network-metrics-daemon-bw29p" (UID: "03fbffc5-9471-44da-b5b0-4b881b17898c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:34.880534 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.880481 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret podName:f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:35.880463446 +0000 UTC m=+4.107139772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret") pod "global-pull-secret-syncer-jphcv" (UID: "f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:34.880534 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.880417 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:34.880534 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.880504 2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5nml for pod openshift-network-diagnostics/network-check-target-rn9px: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:34.880906 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:34.880553 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml podName:32321d1e-f561-4a92-85a2-19420d4d2220 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:35.880540274 +0000 UTC m=+4.107216612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5nml" (UniqueName: "kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml") pod "network-check-target-rn9px" (UID: "32321d1e-f561-4a92-85a2-19420d4d2220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:35.088492 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:35.088455 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e112ec_e721_4ded_a94d_f259b4e2e2b8.slice/crio-daab9f986ec2de14d65e078ca6b880d3354068137dea556e506f6cf7c40420fe WatchSource:0}: Error finding container daab9f986ec2de14d65e078ca6b880d3354068137dea556e506f6cf7c40420fe: Status 404 returned error can't find the container with id daab9f986ec2de14d65e078ca6b880d3354068137dea556e506f6cf7c40420fe Sep 11 18:15:35.089773 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:35.089731 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda245d879_6ccd_4044_b460_19db9dfdc44d.slice/crio-b75e71d5b9583f96f2d7035b02f7a7cfafeb8ff9f51f685dfd8de7abf354bc96 WatchSource:0}: Error finding container b75e71d5b9583f96f2d7035b02f7a7cfafeb8ff9f51f685dfd8de7abf354bc96: Status 404 returned error can't find the container with id b75e71d5b9583f96f2d7035b02f7a7cfafeb8ff9f51f685dfd8de7abf354bc96 Sep 11 18:15:35.093188 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:35.093113 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2f026a_4049_47cb_a43b_7eb14472f1ef.slice/crio-efa7f5ea7206d96e5a9bdddf1e0351adfb6841d374847a7a9fe444318680555f WatchSource:0}: Error finding container efa7f5ea7206d96e5a9bdddf1e0351adfb6841d374847a7a9fe444318680555f: Status 404 returned error can't find the container with id efa7f5ea7206d96e5a9bdddf1e0351adfb6841d374847a7a9fe444318680555f Sep 11 18:15:35.094105 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:35.094088 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b7d068_b3ff_4340_86b4_6919733efb4a.slice/crio-d72e1f1983e6a2f86715f8323aacfdc6bb38d161ee2c1a6dc27236afde321c0c WatchSource:0}: Error finding container d72e1f1983e6a2f86715f8323aacfdc6bb38d161ee2c1a6dc27236afde321c0c: Status 404 returned error can't find the container with id d72e1f1983e6a2f86715f8323aacfdc6bb38d161ee2c1a6dc27236afde321c0c Sep 11 18:15:35.095119 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:35.095097 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fa86ed_523c_4c1b_a59f_3d08c1039f56.slice/crio-1f1ebe4eab0c2ae2928ef1c303788f1e0132eeea1d682b4c7f2ee9fadc61ce49 WatchSource:0}: Error finding container 1f1ebe4eab0c2ae2928ef1c303788f1e0132eeea1d682b4c7f2ee9fadc61ce49: Status 404 returned error can't find the container with id 1f1ebe4eab0c2ae2928ef1c303788f1e0132eeea1d682b4c7f2ee9fadc61ce49 Sep 11 18:15:35.095972 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:35.095939 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a6d135_2518_45ee_b801_709400c2a1a0.slice/crio-5427a4c1f928e3415e17fe60417979005c83eecd08eb8e9aecfd778b8cc42f90 WatchSource:0}: Error finding container 5427a4c1f928e3415e17fe60417979005c83eecd08eb8e9aecfd778b8cc42f90: Status 404 returned error can't find the container with id 5427a4c1f928e3415e17fe60417979005c83eecd08eb8e9aecfd778b8cc42f90 Sep 11 18:15:35.096617 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:35.096600 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377ee3bf_f751_4c9c_9552_eb50fa46d6d5.slice/crio-c1d762f7be239256910617784dfdb5314be10dc1cd67ccc484dfae06fdccf029 WatchSource:0}: Error finding container c1d762f7be239256910617784dfdb5314be10dc1cd67ccc484dfae06fdccf029: Status 404 returned error can't find the container with id c1d762f7be239256910617784dfdb5314be10dc1cd67ccc484dfae06fdccf029 Sep 11 18:15:35.097916 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:35.097864 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb19ff5_33de_41c7_aebf_0756d014e6f3.slice/crio-17f882df9262a558aa7ce767258802f3a8f736cb5ed595fb1b8dac08b7b37b51 WatchSource:0}: Error finding container 17f882df9262a558aa7ce767258802f3a8f736cb5ed595fb1b8dac08b7b37b51: Status 404 returned error can't find the container with id 17f882df9262a558aa7ce767258802f3a8f736cb5ed595fb1b8dac08b7b37b51 Sep 11 18:15:35.100211 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:15:35.100102 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaaf296_872b_4b25_b8da_17a8f77ad1d3.slice/crio-7cfecfe6c1796bbcccf31687a59f0ae95a1bbbf4eec2ec0944a8bd2f3564a6b1 WatchSource:0}: Error finding container 7cfecfe6c1796bbcccf31687a59f0ae95a1bbbf4eec2ec0944a8bd2f3564a6b1: Status 404 returned error can't find the container with id 7cfecfe6c1796bbcccf31687a59f0ae95a1bbbf4eec2ec0944a8bd2f3564a6b1 Sep 11 18:15:35.222820 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.222762 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:34 +0000 UTC" deadline="2028-04-25 23:41:44.290656648 +0000 UTC" Sep 11 18:15:35.222820 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.222815 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14213h26m9.067845305s" Sep 11 18:15:35.233326 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.233289 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal" event={"ID":"6b275f54185b5d5dddefd04d62d766a8","Type":"ContainerStarted","Data":"510dc1e1d48f23005fcfbe86fa960667b25955f43ab6a32dc538b1d1f9e9ac0d"} Sep 11 18:15:35.234483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.234458 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkmbt" event={"ID":"cbaaf296-872b-4b25-b8da-17a8f77ad1d3","Type":"ContainerStarted","Data":"7cfecfe6c1796bbcccf31687a59f0ae95a1bbbf4eec2ec0944a8bd2f3564a6b1"} Sep 11 18:15:35.236645 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.236627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kj7zc" event={"ID":"cdb19ff5-33de-41c7-aebf-0756d014e6f3","Type":"ContainerStarted","Data":"17f882df9262a558aa7ce767258802f3a8f736cb5ed595fb1b8dac08b7b37b51"} Sep 11 18:15:35.237748 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.237726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerStarted","Data":"c1d762f7be239256910617784dfdb5314be10dc1cd67ccc484dfae06fdccf029"} Sep 11 18:15:35.239360 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.239337 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdttd" event={"ID":"e6b7d068-b3ff-4340-86b4-6919733efb4a","Type":"ContainerStarted","Data":"d72e1f1983e6a2f86715f8323aacfdc6bb38d161ee2c1a6dc27236afde321c0c"} Sep 11 18:15:35.241211 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.241163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" event={"ID":"ae2f026a-4049-47cb-a43b-7eb14472f1ef","Type":"ContainerStarted","Data":"efa7f5ea7206d96e5a9bdddf1e0351adfb6841d374847a7a9fe444318680555f"} Sep 11 18:15:35.243996 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.243977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cgv47" event={"ID":"34e112ec-e721-4ded-a94d-f259b4e2e2b8","Type":"ContainerStarted","Data":"daab9f986ec2de14d65e078ca6b880d3354068137dea556e506f6cf7c40420fe"} Sep 11 18:15:35.245070 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.245050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mn77k" event={"ID":"27a6d135-2518-45ee-b801-709400c2a1a0","Type":"ContainerStarted","Data":"5427a4c1f928e3415e17fe60417979005c83eecd08eb8e9aecfd778b8cc42f90"} Sep 11 18:15:35.246219 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.246201 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l6ff2" event={"ID":"f8fa86ed-523c-4c1b-a59f-3d08c1039f56","Type":"ContainerStarted","Data":"1f1ebe4eab0c2ae2928ef1c303788f1e0132eeea1d682b4c7f2ee9fadc61ce49"} Sep 11 18:15:35.247488 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.247467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" event={"ID":"a245d879-6ccd-4044-b460-19db9dfdc44d","Type":"ContainerStarted","Data":"b75e71d5b9583f96f2d7035b02f7a7cfafeb8ff9f51f685dfd8de7abf354bc96"} Sep 11 18:15:35.886229 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.886184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:35.886424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.886254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:35.886424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:35.886298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:35.886544 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:35.886440 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:35.886544 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:35.886503 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret podName:f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:37.886485798 +0000 UTC m=+6.113162136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret") pod "global-pull-secret-syncer-jphcv" (UID: "f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:35.886659 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:35.886566 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:35.886659 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:35.886597 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs podName:03fbffc5-9471-44da-b5b0-4b881b17898c nodeName:}" failed. No retries permitted until 2026-09-11 18:15:37.88658618 +0000 UTC m=+6.113262513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs") pod "network-metrics-daemon-bw29p" (UID: "03fbffc5-9471-44da-b5b0-4b881b17898c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:35.886785 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:35.886671 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:35.886785 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:35.886685 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:35.886785 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:35.886696 2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5nml for pod openshift-network-diagnostics/network-check-target-rn9px: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:35.886785 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:35.886749 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml podName:32321d1e-f561-4a92-85a2-19420d4d2220 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:37.886737297 +0000 UTC m=+6.113413629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5nml" (UniqueName: "kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml") pod "network-check-target-rn9px" (UID: "32321d1e-f561-4a92-85a2-19420d4d2220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:36.226045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:36.224839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:36.226045 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:36.224971 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:36.226045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:36.225370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:36.226045 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:36.225455 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:36.226045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:36.225512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:36.226045 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:36.225577 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:36.226045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:36.225931 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:34 +0000 UTC" deadline="2028-05-22 20:48:42.727226244 +0000 UTC" Sep 11 18:15:36.226045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:36.225954 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14858h33m6.501275688s" Sep 11 18:15:36.279757 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:36.279699 2579 generic.go:356] "Generic (PLEG): container finished" podID="5d99a6b4b5bc4fea990e0ac691bcb542" containerID="b3f1ee3759f96476d9c542aa0d5b1731d4aa1fd3c3a29ea294643fb89a8c2d65" exitCode=0 Sep 11 18:15:36.280320 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:36.280296 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" event={"ID":"5d99a6b4b5bc4fea990e0ac691bcb542","Type":"ContainerDied","Data":"b3f1ee3759f96476d9c542aa0d5b1731d4aa1fd3c3a29ea294643fb89a8c2d65"} Sep 11 18:15:36.296793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:36.296422 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-253.ec2.internal" podStartSLOduration=3.296398531 podStartE2EDuration="3.296398531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:35.248935473 +0000 UTC m=+3.475611813" watchObservedRunningTime="2026-09-11 18:15:36.296398531 +0000 UTC m=+4.523074849" Sep 11 18:15:37.295044 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:37.294768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" event={"ID":"5d99a6b4b5bc4fea990e0ac691bcb542","Type":"ContainerStarted","Data":"dd11b385478c913c0f07bdd87f7d814c22063769e0cbe4ea57daf43b8d30f8a9"} Sep 11 18:15:37.310642 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:37.310549 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-253.ec2.internal" podStartSLOduration=4.310529649 podStartE2EDuration="4.310529649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:37.310264232 +0000 UTC m=+5.536940608" watchObservedRunningTime="2026-09-11 18:15:37.310529649 +0000 UTC m=+5.537205990" Sep 11 18:15:37.907146 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:37.907106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:37.907333 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:37.907172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:37.907333 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:37.907220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:37.907447 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:37.907339 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:37.907447 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:37.907402 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs podName:03fbffc5-9471-44da-b5b0-4b881b17898c nodeName:}" failed. No retries permitted until 2026-09-11 18:15:41.907383883 +0000 UTC m=+10.134060204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs") pod "network-metrics-daemon-bw29p" (UID: "03fbffc5-9471-44da-b5b0-4b881b17898c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:37.908101 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:37.907868 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:37.908101 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:37.907892 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:37.908101 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:37.907906 2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5nml for pod openshift-network-diagnostics/network-check-target-rn9px: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:37.908101 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:37.907956 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml podName:32321d1e-f561-4a92-85a2-19420d4d2220 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:41.907938056 +0000 UTC m=+10.134614375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5nml" (UniqueName: "kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml") pod "network-check-target-rn9px" (UID: "32321d1e-f561-4a92-85a2-19420d4d2220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:37.908101 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:37.908016 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:37.908101 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:37.908048 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret podName:f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:41.908037648 +0000 UTC m=+10.134713979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret") pod "global-pull-secret-syncer-jphcv" (UID: "f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:38.222895 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:38.222814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:38.223054 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:38.222817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:38.223054 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:38.222954 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:38.223054 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:38.223034 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:38.223211 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:38.223094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:38.223211 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:38.223176 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:40.222830 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:40.222799 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:40.223300 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:40.222944 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:40.223365 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:40.223302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:40.223410 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:40.223387 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:40.224355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:40.224329 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:40.224467 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:40.224440 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:41.942226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:41.942288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:41.942324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:41.942416 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:41.942474 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs podName:03fbffc5-9471-44da-b5b0-4b881b17898c nodeName:}" failed. No retries permitted until 2026-09-11 18:15:49.942455662 +0000 UTC m=+18.169131983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs") pod "network-metrics-daemon-bw29p" (UID: "03fbffc5-9471-44da-b5b0-4b881b17898c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:41.942512 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:41.942548 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret podName:f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:49.942538316 +0000 UTC m=+18.169214649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret") pod "global-pull-secret-syncer-jphcv" (UID: "f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:41.942553 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:41.942584 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:41.942598 2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5nml for pod openshift-network-diagnostics/network-check-target-rn9px: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:41.942769 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:41.942655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml podName:32321d1e-f561-4a92-85a2-19420d4d2220 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:49.942638942 +0000 UTC m=+18.169315278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5nml" (UniqueName: "kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml") pod "network-check-target-rn9px" (UID: "32321d1e-f561-4a92-85a2-19420d4d2220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:42.225356 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:42.225284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:42.225622 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:42.225318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:42.225622 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:42.225413 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:42.225810 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:42.225721 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:42.225864 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:42.225811 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:42.225986 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:42.225956 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:44.223325 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:44.223287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:44.223325 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:44.223311 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:44.223882 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:44.223287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:44.223882 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:44.223424 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:44.223882 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:44.223520 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:44.223882 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:44.223673 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:46.222863 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:46.222827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:46.222863 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:46.222864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:46.223395 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:46.222846 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:46.223395 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:46.222946 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:46.223395 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:46.223034 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:46.223395 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:46.223151 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:48.223085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:48.223052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:48.223546 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:48.223050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:48.223546 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:48.223178 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:48.223546 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:48.223263 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:48.223546 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:48.223050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:48.223546 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:48.223356 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:49.993626 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:49.993588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:49.993626 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:49.993642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:49.994264 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:49.993683 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:49.994264 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:49.993775 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:49.994264 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:49.993800 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:49.994264 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:49.993801 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:49.994264 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:49.993812 2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5nml for pod openshift-network-diagnostics/network-check-target-rn9px: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:49.994264 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:49.993863 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs podName:03fbffc5-9471-44da-b5b0-4b881b17898c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:05.993847813 +0000 UTC m=+34.220524132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs") pod "network-metrics-daemon-bw29p" (UID: "03fbffc5-9471-44da-b5b0-4b881b17898c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:49.994264 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:49.993804 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:49.994264 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:49.993883 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml podName:32321d1e-f561-4a92-85a2-19420d4d2220 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:05.993873757 +0000 UTC m=+34.220550076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5nml" (UniqueName: "kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml") pod "network-check-target-rn9px" (UID: "32321d1e-f561-4a92-85a2-19420d4d2220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:49.994264 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:49.993946 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret podName:f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:05.993931484 +0000 UTC m=+34.220607807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret") pod "global-pull-secret-syncer-jphcv" (UID: "f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:50.222438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:50.222401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:50.222613 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:50.222444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:50.222613 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:50.222401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:50.222613 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:50.222531 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:50.222613 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:50.222593 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:50.222835 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:50.222685 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:52.223534 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:52.223504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:52.223534 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:52.223531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:52.224052 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:52.223637 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:52.224052 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:52.223731 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:52.224052 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:52.223755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:52.224052 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:52.223861 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:53.329480 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.329252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" event={"ID":"ae2f026a-4049-47cb-a43b-7eb14472f1ef","Type":"ContainerStarted","Data":"cbfdb68a2c1919cecefc9fc85370638abab0129073eabf112eb801bf4b1d36ae"} Sep 11 18:15:53.330741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.330691 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cgv47" event={"ID":"34e112ec-e721-4ded-a94d-f259b4e2e2b8","Type":"ContainerStarted","Data":"8fdd0587865d9a51950833dfc39e978609fdc3bc964e8c3372885f8e2690e2bf"} Sep 11 18:15:53.332085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.332064 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mn77k" event={"ID":"27a6d135-2518-45ee-b801-709400c2a1a0","Type":"ContainerStarted","Data":"6bd38253c239fa6fc186d9b7412bacd6bc17b08e19f2be8f98f75a65c724926c"} Sep 11 18:15:53.333768 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.333744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" event={"ID":"a245d879-6ccd-4044-b460-19db9dfdc44d","Type":"ContainerStarted","Data":"f6e4aa13768b833159e27a99ed420eadd08446539afe5facaf4f0a8b707e0493"} Sep 11 18:15:53.335209 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.335185 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkmbt" event={"ID":"cbaaf296-872b-4b25-b8da-17a8f77ad1d3","Type":"ContainerStarted","Data":"dff699ceca6a66eef058ec153feeae6d8ea57f8b389102346c24e35c11b08373"} Sep 11 18:15:53.337186 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.337167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerStarted","Data":"604b2d76883ecc7bf58f940bda321c3d085ab1707916e1fc3d8ed1c0bc4677c6"} Sep 11 18:15:53.338632 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.338614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdttd" event={"ID":"e6b7d068-b3ff-4340-86b4-6919733efb4a","Type":"ContainerStarted","Data":"81cdb11370b47d89631b885c376372896e4cb0e47f362fcb15b3c729025cbdad"} Sep 11 18:15:53.344574 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.344542 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bn5pf" podStartSLOduration=3.434637419 podStartE2EDuration="21.344532491s" totalImagesPullingTime="17.909895072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:53.344440814 +0000 UTC m=+21.571117173" watchObservedRunningTime="2026-09-11 18:15:53.344532491 +0000 UTC m=+21.571208830" Sep 11 18:15:53.357480 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.357434 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mn77k" podStartSLOduration=3.475971119 podStartE2EDuration="21.357415828s" totalImagesPullingTime="17.881444709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:53.356850128 +0000 UTC m=+21.583526492" watchObservedRunningTime="2026-09-11 18:15:53.357415828 +0000 UTC m=+21.584092169" Sep 11 18:15:53.373953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.373930 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:53.374618 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.374603 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:53.395574 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.395519 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wkmbt" podStartSLOduration=3.55084562 podStartE2EDuration="21.395499601s" totalImagesPullingTime="17.844653981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:53.395260913 +0000 UTC m=+21.621937253" watchObservedRunningTime="2026-09-11 18:15:53.395499601 +0000 UTC m=+21.622175942" Sep 11 18:15:53.417430 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:53.417389 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cgv47" podStartSLOduration=7.4254285079999995 podStartE2EDuration="21.417375884s" totalImagesPullingTime="13.991947376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:53.415938297 +0000 UTC m=+21.642614636" watchObservedRunningTime="2026-09-11 18:15:53.417375884 +0000 UTC m=+21.644052202" Sep 11 18:15:54.222629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.222455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:54.222818 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.222455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:54.222818 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:54.222730 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:54.222818 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:54.222808 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:54.223031 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.222455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:54.223031 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:54.222913 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:54.342396 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.342369 2579 generic.go:356] "Generic (PLEG): container finished" podID="e6b7d068-b3ff-4340-86b4-6919733efb4a" containerID="81cdb11370b47d89631b885c376372896e4cb0e47f362fcb15b3c729025cbdad" exitCode=0 Sep 11 18:15:54.343112 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.342452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdttd" event={"ID":"e6b7d068-b3ff-4340-86b4-6919733efb4a","Type":"ContainerDied","Data":"81cdb11370b47d89631b885c376372896e4cb0e47f362fcb15b3c729025cbdad"} Sep 11 18:15:54.343923 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.343905 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l6ff2" event={"ID":"f8fa86ed-523c-4c1b-a59f-3d08c1039f56","Type":"ContainerStarted","Data":"3bed48f8993a314937bbba9fc0ed2efd42469b7cfe9682682e6d067e1c7f21a9"} Sep 11 18:15:54.345394 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.345366 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kj7zc" event={"ID":"cdb19ff5-33de-41c7-aebf-0756d014e6f3","Type":"ContainerStarted","Data":"39a948b3e485b3733b89ebf7db7d87e073a3426a253c3df86070b5a97637b4b1"} Sep 11 18:15:54.348044 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.348027 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:15:54.348327 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.348309 2579 generic.go:356] "Generic (PLEG): container finished" podID="377ee3bf-f751-4c9c-9552-eb50fa46d6d5" containerID="ded9bdd25ff71996bdb829ba82fde896c82d16de4d38d8f0a71c5eaf85df3d57" exitCode=1 Sep 11 18:15:54.348409 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.348389 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerStarted","Data":"22325a63f1fed25c54ef9ed5bf64e980e5a263f3be74fc2e3c00446dc23912a0"} Sep 11 18:15:54.348456 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.348448 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerStarted","Data":"58af790f3666c0deb129f8860f9edcd04cfea7e567cae8c2768302443f520ce9"} Sep 11 18:15:54.348496 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.348486 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerStarted","Data":"a78c79d4586957f1b720fbf7f7e93924a5bc16549691ec4b3f790cf3b37147ea"} Sep 11 18:15:54.348535 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.348525 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerStarted","Data":"e2ae90a2e5eb74b5d18ef80c954895887908976abe5e467ff9017ae6ef01bec3"} Sep 11 18:15:54.348582 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.348567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerDied","Data":"ded9bdd25ff71996bdb829ba82fde896c82d16de4d38d8f0a71c5eaf85df3d57"} Sep 11 18:15:54.394787 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.394750 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l6ff2" podStartSLOduration=4.486458815 podStartE2EDuration="22.3947342s" totalImagesPullingTime="17.908275385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:54.393976931 +0000 UTC m=+22.620653290" watchObservedRunningTime="2026-09-11 18:15:54.3947342 +0000 UTC m=+22.621410539" Sep 11 18:15:54.518073 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:54.518048 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 18:15:55.219554 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:55.219449 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:15:54.51806738Z","UUID":"08fa7992-f9d4-435d-8b9d-8df2922fd391","Handler":null,"Name":"","Endpoint":""} Sep 11 18:15:55.221370 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:55.221347 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 18:15:55.221497 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:55.221387 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 18:15:55.353197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:55.353152 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" event={"ID":"a245d879-6ccd-4044-b460-19db9dfdc44d","Type":"ContainerStarted","Data":"977e75931951490351c08365a41b1dfbcc16cb7e399ad809f54e5bfcf81090b0"} Sep 11 18:15:55.353628 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:55.353237 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 18:15:56.222903 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:56.222868 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:56.223086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:56.222918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:56.223086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:56.222868 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:56.223086 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:56.223002 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:56.223259 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:56.223079 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:56.223259 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:56.223140 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:56.358245 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:56.358209 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" event={"ID":"a245d879-6ccd-4044-b460-19db9dfdc44d","Type":"ContainerStarted","Data":"2eeb1e3df78f4101505893138010b32d9c3a4ac4773b8daa8076eafe71d61db8"} Sep 11 18:15:56.361919 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:56.361895 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:15:56.362329 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:56.362306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerStarted","Data":"4ddeae3a8ed0daefde0802b931321ed466a89f292d94081dd315906dd3b99d7a"} Sep 11 18:15:56.376604 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:56.376554 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsgsv" podStartSLOduration=4.377063362 podStartE2EDuration="24.376540429s" totalImagesPullingTime="19.999477067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:56.376418377 +0000 UTC m=+24.603094737" watchObservedRunningTime="2026-09-11 18:15:56.376540429 +0000 UTC m=+24.603216768" Sep 11 18:15:56.377088 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:56.377064 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kj7zc" podStartSLOduration=6.0877103 podStartE2EDuration="24.37705821s" totalImagesPullingTime="18.28934791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:54.467954474 +0000 UTC m=+22.694630830" watchObservedRunningTime="2026-09-11 18:15:56.37705821 +0000 UTC m=+24.603734550" Sep 11 18:15:58.226164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:58.226086 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:15:58.226600 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:58.226207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:15:58.226600 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:58.226236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:15:58.226600 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:58.226228 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:15:58.226600 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:58.226316 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:15:58.226600 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:15:58.226396 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:15:59.372573 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.372396 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:15:59.373283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.372950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerStarted","Data":"89d158a1518c9a92a8967e1ae73cc1eec8459273d6ae8856092229be4c21647e"} Sep 11 18:15:59.373283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.373157 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:59.373440 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.373421 2579 scope.go:117] "RemoveContainer" containerID="ded9bdd25ff71996bdb829ba82fde896c82d16de4d38d8f0a71c5eaf85df3d57" Sep 11 18:15:59.374910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.374889 2579 generic.go:356] "Generic (PLEG): container finished" podID="e6b7d068-b3ff-4340-86b4-6919733efb4a" containerID="87d0fd37769a4e29f0fbb9e82e2b284d0fcf069a69e6278f90f130eab5a7ef47" exitCode=0 Sep 11 18:15:59.375014 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.374927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdttd" event={"ID":"e6b7d068-b3ff-4340-86b4-6919733efb4a","Type":"ContainerDied","Data":"87d0fd37769a4e29f0fbb9e82e2b284d0fcf069a69e6278f90f130eab5a7ef47"} Sep 11 18:15:59.389416 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.389392 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:15:59.791046 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.790140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:15:59.791046 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.790483 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 18:15:59.791875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:15:59.791853 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cgv47" Sep 11 18:16:00.222993 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.222967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:16:00.223115 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.222973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:00.223115 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:00.223071 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:16:00.223223 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.222980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:16:00.223223 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:00.223150 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:16:00.223223 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:00.223194 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:16:00.380541 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.380511 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:16:00.381075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.380913 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" event={"ID":"377ee3bf-f751-4c9c-9552-eb50fa46d6d5","Type":"ContainerStarted","Data":"fbf16a5c9eeb9bd59388d60a95f6e0828078a441f21fc1f97d329e3d8368654c"} Sep 11 18:16:00.381139 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.381122 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:16:00.381288 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.381270 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:16:00.383136 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.383111 2579 generic.go:356] "Generic (PLEG): container finished" podID="e6b7d068-b3ff-4340-86b4-6919733efb4a" containerID="a092ae893d20dbf91a4fc271ce9bdeed36caca8c8ea2d4793a35dc281f588568" exitCode=0 Sep 11 18:16:00.383265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.383216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdttd" event={"ID":"e6b7d068-b3ff-4340-86b4-6919733efb4a","Type":"ContainerDied","Data":"a092ae893d20dbf91a4fc271ce9bdeed36caca8c8ea2d4793a35dc281f588568"} Sep 11 18:16:00.396886 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.396828 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rn9px"] Sep 11 18:16:00.396989 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.396921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:00.397035 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:00.397010 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:16:00.398020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.398002 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:16:00.400336 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.400313 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jphcv"] Sep 11 18:16:00.400430 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.400405 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:16:00.400518 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:00.400495 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:16:00.400897 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.400876 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bw29p"] Sep 11 18:16:00.400986 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.400951 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:16:00.401044 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:00.401030 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:16:00.415309 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:00.415250 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" podStartSLOduration=10.496069133 podStartE2EDuration="28.415232212s" totalImagesPullingTime="17.919163079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:00.415075713 +0000 UTC m=+28.641752071" watchObservedRunningTime="2026-09-11 18:16:00.415232212 +0000 UTC m=+28.641908553" Sep 11 18:16:01.388241 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:01.387952 2579 generic.go:356] "Generic (PLEG): container finished" podID="e6b7d068-b3ff-4340-86b4-6919733efb4a" containerID="6f38de31757bb4f44340292f26208d37ee916d008ca151e7c20372f34d08a7eb" exitCode=0 Sep 11 18:16:01.388650 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:01.388020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdttd" event={"ID":"e6b7d068-b3ff-4340-86b4-6919733efb4a","Type":"ContainerDied","Data":"6f38de31757bb4f44340292f26208d37ee916d008ca151e7c20372f34d08a7eb"} Sep 11 18:16:02.224170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:02.223777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:16:02.224170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:02.223889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:02.224170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:02.223932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:16:02.224170 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:02.223929 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:16:02.224170 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:02.224019 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:16:02.224170 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:02.224084 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:16:04.222482 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:04.222405 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:04.222482 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:04.222419 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:16:04.222482 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:04.222404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:16:04.223043 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:04.222530 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rn9px" podUID="32321d1e-f561-4a92-85a2-19420d4d2220" Sep 11 18:16:04.223043 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:04.222661 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:16:04.223043 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:04.222760 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jphcv" podUID="f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3" Sep 11 18:16:06.008042 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.008009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.008057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.008091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.008178 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.008192 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.008209 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.008220 2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5nml for pod openshift-network-diagnostics/network-check-target-rn9px: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.008178 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.008247 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret podName:f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:38.008228641 +0000 UTC m=+66.234904981 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret") pod "global-pull-secret-syncer-jphcv" (UID: "f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.008264 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml podName:32321d1e-f561-4a92-85a2-19420d4d2220 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:38.008255235 +0000 UTC m=+66.234931559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5nml" (UniqueName: "kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml") pod "network-check-target-rn9px" (UID: "32321d1e-f561-4a92-85a2-19420d4d2220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:16:06.008511 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.008279 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs podName:03fbffc5-9471-44da-b5b0-4b881b17898c nodeName:}" failed. No retries permitted until 2026-09-11 18:16:38.008270683 +0000 UTC m=+66.234947000 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs") pod "network-metrics-daemon-bw29p" (UID: "03fbffc5-9471-44da-b5b0-4b881b17898c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:16:06.034312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.034289 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-253.ec2.internal" event="NodeReady" Sep 11 18:16:06.034419 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.034412 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 18:16:06.068097 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.068072 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-648d6bd57b-sljfs"] Sep 11 18:16:06.082608 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.082587 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bhn27"] Sep 11 18:16:06.082744 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.082723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.087426 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.087239 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 18:16:06.087426 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.087291 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 18:16:06.087426 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.087302 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 18:16:06.087822 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.087612 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-grmtt\"" Sep 11 18:16:06.091036 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.091016 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648d6bd57b-sljfs"] Sep 11 18:16:06.091142 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.091045 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4f5s2"] Sep 11 18:16:06.091142 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.091139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.093805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.093786 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 18:16:06.093949 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.093932 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 18:16:06.093999 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.093935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksdw9\"" Sep 11 18:16:06.094048 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.094016 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 18:16:06.108942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.108880 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bhn27"] Sep 11 18:16:06.108942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.108933 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4f5s2"] Sep 11 18:16:06.109081 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.109052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4f5s2" Sep 11 18:16:06.111409 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.111390 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 18:16:06.111605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.111542 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 18:16:06.111660 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.111638 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 18:16:06.111803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.111788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f8nfl\"" Sep 11 18:16:06.209184 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.209184 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-registry-certificates\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.209387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/050a7f57-b326-4310-8617-5075add177fb-ca-trust-extracted\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.209387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvp5\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-kube-api-access-7vvp5\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.209387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0524f5b1-df3e-4288-8864-709be1f0f4e4-cert\") pod \"ingress-canary-4f5s2\" (UID: \"0524f5b1-df3e-4288-8864-709be1f0f4e4\") " pod="openshift-ingress-canary/ingress-canary-4f5s2" Sep 11 18:16:06.209387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-config-volume\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.209387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.209579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-trusted-ca\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.209579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-tmp-dir\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.209579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwk8x\" (UniqueName: \"kubernetes.io/projected/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-kube-api-access-mwk8x\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.209579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-bound-sa-token\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.209579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlmpc\" (UniqueName: \"kubernetes.io/projected/0524f5b1-df3e-4288-8864-709be1f0f4e4-kube-api-access-nlmpc\") pod \"ingress-canary-4f5s2\" (UID: \"0524f5b1-df3e-4288-8864-709be1f0f4e4\") " pod="openshift-ingress-canary/ingress-canary-4f5s2" Sep 11 18:16:06.209579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-image-registry-private-configuration\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.209579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.209573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-installation-pull-secrets\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.223333 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.223304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:16:06.223450 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.223338 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:06.223450 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.223304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:16:06.226183 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.226164 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 18:16:06.226183 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.226171 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kq7d5\"" Sep 11 18:16:06.226382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.226191 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 18:16:06.226382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.226210 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 18:16:06.226382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.226171 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhxbh\"" Sep 11 18:16:06.226382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.226268 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 18:16:06.310864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.310829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.310864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.310864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-registry-certificates\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.310881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/050a7f57-b326-4310-8617-5075add177fb-ca-trust-extracted\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.310897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvp5\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-kube-api-access-7vvp5\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.310912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0524f5b1-df3e-4288-8864-709be1f0f4e4-cert\") pod \"ingress-canary-4f5s2\" (UID: \"0524f5b1-df3e-4288-8864-709be1f0f4e4\") " pod="openshift-ingress-canary/ingress-canary-4f5s2" Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.310941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-config-volume\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.310967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.311041 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-trusted-ca\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-tmp-dir\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.311124 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls podName:7c2e0dbc-ad02-49f5-8604-4a45cfba2237 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:06.811102488 +0000 UTC m=+35.037778809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls") pod "dns-default-bhn27" (UID: "7c2e0dbc-ad02-49f5-8604-4a45cfba2237") : secret "dns-default-metrics-tls" not found Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.311129 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:16:06.311141 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.311144 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d6bd57b-sljfs: secret "image-registry-tls" not found Sep 11 18:16:06.311605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk8x\" (UniqueName: \"kubernetes.io/projected/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-kube-api-access-mwk8x\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.311605 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.311188 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls podName:050a7f57-b326-4310-8617-5075add177fb nodeName:}" failed. No retries permitted until 2026-09-11 18:16:06.811175948 +0000 UTC m=+35.037852282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls") pod "image-registry-648d6bd57b-sljfs" (UID: "050a7f57-b326-4310-8617-5075add177fb") : secret "image-registry-tls" not found Sep 11 18:16:06.311605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-bound-sa-token\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.311605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmpc\" (UniqueName: \"kubernetes.io/projected/0524f5b1-df3e-4288-8864-709be1f0f4e4-kube-api-access-nlmpc\") pod \"ingress-canary-4f5s2\" (UID: \"0524f5b1-df3e-4288-8864-709be1f0f4e4\") " pod="openshift-ingress-canary/ingress-canary-4f5s2" Sep 11 18:16:06.311605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-image-registry-private-configuration\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.311605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-installation-pull-secrets\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.311605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-tmp-dir\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.311979 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/050a7f57-b326-4310-8617-5075add177fb-ca-trust-extracted\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.311979 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.311846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-config-volume\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.312193 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.312034 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-registry-certificates\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.312455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.312435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-trusted-ca\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.315275 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.315248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-installation-pull-secrets\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.315439 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.315418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0524f5b1-df3e-4288-8864-709be1f0f4e4-cert\") pod \"ingress-canary-4f5s2\" (UID: \"0524f5b1-df3e-4288-8864-709be1f0f4e4\") " pod="openshift-ingress-canary/ingress-canary-4f5s2" Sep 11 18:16:06.317167 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.317146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-image-registry-private-configuration\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.322822 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.322802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlmpc\" (UniqueName: \"kubernetes.io/projected/0524f5b1-df3e-4288-8864-709be1f0f4e4-kube-api-access-nlmpc\") pod \"ingress-canary-4f5s2\" (UID: \"0524f5b1-df3e-4288-8864-709be1f0f4e4\") " pod="openshift-ingress-canary/ingress-canary-4f5s2" Sep 11 18:16:06.322941 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.322878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-bound-sa-token\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.323131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.323111 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvp5\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-kube-api-access-7vvp5\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.323327 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.323311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwk8x\" (UniqueName: \"kubernetes.io/projected/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-kube-api-access-mwk8x\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.402358 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.402272 2579 generic.go:356] "Generic (PLEG): container finished" podID="e6b7d068-b3ff-4340-86b4-6919733efb4a" containerID="bcc2a47531b2de44eabb816f842e25aeed46a311832c148f288eb3153593c53b" exitCode=0 Sep 11 18:16:06.402358 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.402331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdttd" event={"ID":"e6b7d068-b3ff-4340-86b4-6919733efb4a","Type":"ContainerDied","Data":"bcc2a47531b2de44eabb816f842e25aeed46a311832c148f288eb3153593c53b"} Sep 11 18:16:06.416953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.416932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4f5s2" Sep 11 18:16:06.729437 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.729368 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4f5s2"] Sep 11 18:16:06.733069 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:16:06.733029 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0524f5b1_df3e_4288_8864_709be1f0f4e4.slice/crio-ca0414bfeee552903e75195c95c52ff6ba0f5c3635060d5eaa8beba2bc0c9474 WatchSource:0}: Error finding container ca0414bfeee552903e75195c95c52ff6ba0f5c3635060d5eaa8beba2bc0c9474: Status 404 returned error can't find the container with id ca0414bfeee552903e75195c95c52ff6ba0f5c3635060d5eaa8beba2bc0c9474 Sep 11 18:16:06.815682 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.815649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:06.815878 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:06.815696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:06.815878 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.815810 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:06.815878 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.815871 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls podName:7c2e0dbc-ad02-49f5-8604-4a45cfba2237 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:07.815857294 +0000 UTC m=+36.042533617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls") pod "dns-default-bhn27" (UID: "7c2e0dbc-ad02-49f5-8604-4a45cfba2237") : secret "dns-default-metrics-tls" not found Sep 11 18:16:06.815999 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.815813 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:16:06.815999 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.815917 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d6bd57b-sljfs: secret "image-registry-tls" not found Sep 11 18:16:06.815999 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:06.815959 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls podName:050a7f57-b326-4310-8617-5075add177fb nodeName:}" failed. No retries permitted until 2026-09-11 18:16:07.815947973 +0000 UTC m=+36.042624293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls") pod "image-registry-648d6bd57b-sljfs" (UID: "050a7f57-b326-4310-8617-5075add177fb") : secret "image-registry-tls" not found Sep 11 18:16:07.407002 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:07.406967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4f5s2" event={"ID":"0524f5b1-df3e-4288-8864-709be1f0f4e4","Type":"ContainerStarted","Data":"ca0414bfeee552903e75195c95c52ff6ba0f5c3635060d5eaa8beba2bc0c9474"} Sep 11 18:16:07.410521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:07.410489 2579 generic.go:356] "Generic (PLEG): container finished" podID="e6b7d068-b3ff-4340-86b4-6919733efb4a" containerID="38ea44c47b71d29ee5ddc69fbe4a3aaa80a0539e6bf3e9adcabaf724ab7c4729" exitCode=0 Sep 11 18:16:07.410677 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:07.410541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdttd" event={"ID":"e6b7d068-b3ff-4340-86b4-6919733efb4a","Type":"ContainerDied","Data":"38ea44c47b71d29ee5ddc69fbe4a3aaa80a0539e6bf3e9adcabaf724ab7c4729"} Sep 11 18:16:07.823197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:07.822961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:07.823376 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:07.823229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:07.823376 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:07.823121 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:07.823376 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:07.823343 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls podName:7c2e0dbc-ad02-49f5-8604-4a45cfba2237 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:09.823318201 +0000 UTC m=+38.049994533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls") pod "dns-default-bhn27" (UID: "7c2e0dbc-ad02-49f5-8604-4a45cfba2237") : secret "dns-default-metrics-tls" not found Sep 11 18:16:07.823376 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:07.823367 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:16:07.823585 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:07.823382 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d6bd57b-sljfs: secret "image-registry-tls" not found Sep 11 18:16:07.823585 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:07.823429 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls podName:050a7f57-b326-4310-8617-5075add177fb nodeName:}" failed. No retries permitted until 2026-09-11 18:16:09.823414651 +0000 UTC m=+38.050090988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls") pod "image-registry-648d6bd57b-sljfs" (UID: "050a7f57-b326-4310-8617-5075add177fb") : secret "image-registry-tls" not found Sep 11 18:16:08.417387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:08.417351 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdttd" event={"ID":"e6b7d068-b3ff-4340-86b4-6919733efb4a","Type":"ContainerStarted","Data":"1a7a91c448594396b14cbc29bf5e0899ac21a2c86f9cbaf95fc2569cf9e59342"} Sep 11 18:16:08.438755 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:08.438695 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kdttd" podStartSLOduration=8.549977669 podStartE2EDuration="36.438681049s" totalImagesPullingTime="27.88870338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:08.437673581 +0000 UTC m=+36.664349940" watchObservedRunningTime="2026-09-11 18:16:08.438681049 +0000 UTC m=+36.665357389" Sep 11 18:16:08.584754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:08.584720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:08.686454 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:08.686333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:09.421760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:09.421716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4f5s2" event={"ID":"0524f5b1-df3e-4288-8864-709be1f0f4e4","Type":"ContainerStarted","Data":"59bba814dcb7274a6a5bb8737dfc7cd64e79e31825d0ce922fe28bc48bc0ca90"} Sep 11 18:16:09.444642 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:09.444595 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4f5s2" podStartSLOduration=0.947929578 podStartE2EDuration="3.444576666s" totalImagesPullingTime="2.496647088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:09.443306083 +0000 UTC m=+37.669982422" watchObservedRunningTime="2026-09-11 18:16:09.444576666 +0000 UTC m=+37.671253005" Sep 11 18:16:09.840966 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:09.840931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:09.841123 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:09.840982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:09.841123 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:09.841074 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:16:09.841123 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:09.841075 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:09.841277 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:09.841138 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls podName:7c2e0dbc-ad02-49f5-8604-4a45cfba2237 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:13.841123439 +0000 UTC m=+42.067799757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls") pod "dns-default-bhn27" (UID: "7c2e0dbc-ad02-49f5-8604-4a45cfba2237") : secret "dns-default-metrics-tls" not found Sep 11 18:16:09.841277 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:09.841085 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d6bd57b-sljfs: secret "image-registry-tls" not found Sep 11 18:16:09.841277 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:09.841224 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls podName:050a7f57-b326-4310-8617-5075add177fb nodeName:}" failed. No retries permitted until 2026-09-11 18:16:13.841210466 +0000 UTC m=+42.067886786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls") pod "image-registry-648d6bd57b-sljfs" (UID: "050a7f57-b326-4310-8617-5075add177fb") : secret "image-registry-tls" not found Sep 11 18:16:11.647096 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:11.647034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:11.747939 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:11.747898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:13.317793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:13.317760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:13.390688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:13.390657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:13.874190 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:13.874143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:13.874367 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:13.874201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:13.874367 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:13.874291 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:16:13.874367 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:13.874293 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:13.874367 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:13.874302 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d6bd57b-sljfs: secret "image-registry-tls" not found Sep 11 18:16:13.874367 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:13.874360 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls podName:7c2e0dbc-ad02-49f5-8604-4a45cfba2237 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:21.874344601 +0000 UTC m=+50.101020919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls") pod "dns-default-bhn27" (UID: "7c2e0dbc-ad02-49f5-8604-4a45cfba2237") : secret "dns-default-metrics-tls" not found Sep 11 18:16:13.874534 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:13.874404 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls podName:050a7f57-b326-4310-8617-5075add177fb nodeName:}" failed. No retries permitted until 2026-09-11 18:16:21.874384775 +0000 UTC m=+50.101061100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls") pod "image-registry-648d6bd57b-sljfs" (UID: "050a7f57-b326-4310-8617-5075add177fb") : secret "image-registry-tls" not found Sep 11 18:16:15.347360 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:15.347330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:15.447577 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:15.447548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:15.836909 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:15.836879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:15.894169 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:15.894130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:17.281510 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:17.281475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:17.382000 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:17.381968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:21.529041 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:21.529002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:21.630769 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:21.630729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:21.934641 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:21.934553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:21.934641 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:21.934603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:21.934838 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:21.934734 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:21.934838 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:21.934812 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls podName:7c2e0dbc-ad02-49f5-8604-4a45cfba2237 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:37.934793419 +0000 UTC m=+66.161469742 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls") pod "dns-default-bhn27" (UID: "7c2e0dbc-ad02-49f5-8604-4a45cfba2237") : secret "dns-default-metrics-tls" not found Sep 11 18:16:21.934838 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:21.934736 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:16:21.934945 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:21.934841 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d6bd57b-sljfs: secret "image-registry-tls" not found Sep 11 18:16:21.934945 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:21.934896 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls podName:050a7f57-b326-4310-8617-5075add177fb nodeName:}" failed. No retries permitted until 2026-09-11 18:16:37.934884018 +0000 UTC m=+66.161560335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls") pod "image-registry-648d6bd57b-sljfs" (UID: "050a7f57-b326-4310-8617-5075add177fb") : secret "image-registry-tls" not found Sep 11 18:16:32.401279 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:32.401249 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4sfd" Sep 11 18:16:32.563991 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:32.563955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:32.624894 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:32.624858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:36.308024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:36.307984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:36.409504 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:36.409467 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:37.790833 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:37.790795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:37.849898 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:37.849867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:37.945266 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:37.945234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:16:37.945445 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:37.945398 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:16:37.945445 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:37.945422 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d6bd57b-sljfs: secret "image-registry-tls" not found Sep 11 18:16:37.945543 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:37.945487 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls podName:050a7f57-b326-4310-8617-5075add177fb nodeName:}" failed. No retries permitted until 2026-09-11 18:17:09.945468561 +0000 UTC m=+98.172144901 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls") pod "image-registry-648d6bd57b-sljfs" (UID: "050a7f57-b326-4310-8617-5075add177fb") : secret "image-registry-tls" not found Sep 11 18:16:37.945543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:37.945423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:16:37.945543 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:37.945506 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:37.945689 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:37.945548 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls podName:7c2e0dbc-ad02-49f5-8604-4a45cfba2237 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:09.945537555 +0000 UTC m=+98.172213881 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls") pod "dns-default-bhn27" (UID: "7c2e0dbc-ad02-49f5-8604-4a45cfba2237") : secret "dns-default-metrics-tls" not found Sep 11 18:16:38.046061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.045973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:38.046061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.046037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:16:38.046263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.046090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:16:38.049056 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.049039 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 18:16:38.049056 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.049049 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 18:16:38.049358 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.049343 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 18:16:38.056478 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:38.056457 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 18:16:38.056557 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:16:38.056526 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs podName:03fbffc5-9471-44da-b5b0-4b881b17898c nodeName:}" failed. No retries permitted until 2026-09-11 18:17:42.056506048 +0000 UTC m=+130.283182365 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs") pod "network-metrics-daemon-bw29p" (UID: "03fbffc5-9471-44da-b5b0-4b881b17898c") : secret "metrics-daemon-secret" not found Sep 11 18:16:38.059570 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.059553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3-original-pull-secret\") pod \"global-pull-secret-syncer-jphcv\" (UID: \"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3\") " pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:16:38.059990 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.059975 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 18:16:38.068807 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.068788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nml\" (UniqueName: \"kubernetes.io/projected/32321d1e-f561-4a92-85a2-19420d4d2220-kube-api-access-q5nml\") pod \"network-check-target-rn9px\" (UID: \"32321d1e-f561-4a92-85a2-19420d4d2220\") " pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:38.338042 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.337958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jphcv" Sep 11 18:16:38.346620 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.346602 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhxbh\"" Sep 11 18:16:38.353838 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.353815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:38.608840 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.608762 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jphcv"] Sep 11 18:16:38.612008 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:16:38.611983 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d8133f_e8a5_42a2_89e4_8c24af6d2ac3.slice/crio-38a00f7ca7ca2a1e6821d298c5e38e8d6a4eec77b9a05f265cbc0733ba5ff4a2 WatchSource:0}: Error finding container 38a00f7ca7ca2a1e6821d298c5e38e8d6a4eec77b9a05f265cbc0733ba5ff4a2: Status 404 returned error can't find the container with id 38a00f7ca7ca2a1e6821d298c5e38e8d6a4eec77b9a05f265cbc0733ba5ff4a2 Sep 11 18:16:38.632473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:38.632450 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rn9px"] Sep 11 18:16:38.635268 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:16:38.635246 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32321d1e_f561_4a92_85a2_19420d4d2220.slice/crio-6dbc09c3aac303e37084778a830b11e25132eb0b17da3efe8d36294a370b988e WatchSource:0}: Error finding container 6dbc09c3aac303e37084778a830b11e25132eb0b17da3efe8d36294a370b988e: Status 404 returned error can't find the container with id 6dbc09c3aac303e37084778a830b11e25132eb0b17da3efe8d36294a370b988e Sep 11 18:16:39.495885 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:39.495845 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rn9px" event={"ID":"32321d1e-f561-4a92-85a2-19420d4d2220","Type":"ContainerStarted","Data":"6dbc09c3aac303e37084778a830b11e25132eb0b17da3efe8d36294a370b988e"} Sep 11 18:16:39.497382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:39.497346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jphcv" event={"ID":"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3","Type":"ContainerStarted","Data":"38a00f7ca7ca2a1e6821d298c5e38e8d6a4eec77b9a05f265cbc0733ba5ff4a2"} Sep 11 18:16:43.509272 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:43.509229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rn9px" event={"ID":"32321d1e-f561-4a92-85a2-19420d4d2220","Type":"ContainerStarted","Data":"90953811ea6ed9353636dc3ea71dc3e0914c13a3294ba675431106dff88ab900"} Sep 11 18:16:43.509663 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:43.509305 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:16:43.510552 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:43.510530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jphcv" event={"ID":"f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3","Type":"ContainerStarted","Data":"9c7e3682416fc3da23814844f152ac31adc72e21b10e88e3e3064656c1915dc8"} Sep 11 18:16:43.526852 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:43.526798 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rn9px" podStartSLOduration=67.180665628 podStartE2EDuration="1m11.526780005s" totalImagesPullingTime="4.346114377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:43.525587387 +0000 UTC m=+71.752263747" watchObservedRunningTime="2026-09-11 18:16:43.526780005 +0000 UTC m=+71.753456345" Sep 11 18:16:43.543044 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:43.543000 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jphcv" podStartSLOduration=67.165599953 podStartE2EDuration="1m11.542985478s" totalImagesPullingTime="4.377385525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:43.542922536 +0000 UTC m=+71.769598876" watchObservedRunningTime="2026-09-11 18:16:43.542985478 +0000 UTC m=+71.769661853" Sep 11 18:16:52.610012 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:52.609977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:52.711180 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:16:52.711139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:09.980866 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:09.980831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:17:09.981161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:09.980881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:17:09.981161 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:09.980984 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:17:09.981161 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:09.980998 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:17:09.981161 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:09.981010 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d6bd57b-sljfs: secret "image-registry-tls" not found Sep 11 18:17:09.981161 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:09.981060 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls podName:7c2e0dbc-ad02-49f5-8604-4a45cfba2237 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:13.981043861 +0000 UTC m=+162.207720194 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls") pod "dns-default-bhn27" (UID: "7c2e0dbc-ad02-49f5-8604-4a45cfba2237") : secret "dns-default-metrics-tls" not found Sep 11 18:17:09.981161 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:09.981074 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls podName:050a7f57-b326-4310-8617-5075add177fb nodeName:}" failed. No retries permitted until 2026-09-11 18:18:13.981068647 +0000 UTC m=+162.207744965 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls") pod "image-registry-648d6bd57b-sljfs" (UID: "050a7f57-b326-4310-8617-5075add177fb") : secret "image-registry-tls" not found Sep 11 18:17:14.514487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:14.514449 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rn9px" Sep 11 18:17:42.094826 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:42.094764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:17:42.095295 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:42.094929 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 18:17:42.095295 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:42.095006 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs podName:03fbffc5-9471-44da-b5b0-4b881b17898c nodeName:}" failed. No retries permitted until 2026-09-11 18:19:44.094987486 +0000 UTC m=+252.321663805 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs") pod "network-metrics-daemon-bw29p" (UID: "03fbffc5-9471-44da-b5b0-4b881b17898c") : secret "metrics-daemon-secret" not found Sep 11 18:17:44.469697 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.469653 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4"] Sep 11 18:17:44.471387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.471367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:44.473778 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.473747 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 18:17:44.474321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.474295 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 18:17:44.476248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.476227 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tp9fg\"" Sep 11 18:17:44.476644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.476627 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 18:17:44.477027 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.477011 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 18:17:44.493447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.493423 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4"] Sep 11 18:17:44.567905 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.567874 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd"] Sep 11 18:17:44.569761 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.569743 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9"] Sep 11 18:17:44.569868 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.569851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.571692 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.571674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:44.572375 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.572356 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 18:17:44.572642 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.572626 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 18:17:44.573430 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.573413 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:17:44.573509 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.573418 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 18:17:44.573580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.573561 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nrffz\"" Sep 11 18:17:44.574209 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.574192 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 18:17:44.574517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.574503 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 18:17:44.574564 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.574506 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:17:44.574816 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.574800 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lfc58\"" Sep 11 18:17:44.584223 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.584205 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd"] Sep 11 18:17:44.584937 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.584919 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9"] Sep 11 18:17:44.610324 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.610294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bab5ef01-e412-4447-a555-2913d63a81c6-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:44.610451 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.610342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mtw\" (UniqueName: \"kubernetes.io/projected/bab5ef01-e412-4447-a555-2913d63a81c6-kube-api-access-r5mtw\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:44.610531 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.610509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:44.674452 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.674418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:44.711402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.711374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2xhfd\" (UID: \"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.711520 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.711419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2xhfd\" (UID: \"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.711520 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.711510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:44.711643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.711545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwvl5\" (UniqueName: \"kubernetes.io/projected/8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84-kube-api-access-qwvl5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2xhfd\" (UID: \"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.711643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.711618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bab5ef01-e412-4447-a555-2913d63a81c6-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:44.711749 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:44.711644 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:44.711749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.711645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:44.711749 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:44.711725 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls podName:bab5ef01-e412-4447-a555-2913d63a81c6 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:45.211694356 +0000 UTC m=+133.438370679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-jf6v4" (UID: "bab5ef01-e412-4447-a555-2913d63a81c6") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:44.711901 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.711778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mtw\" (UniqueName: \"kubernetes.io/projected/bab5ef01-e412-4447-a555-2913d63a81c6-kube-api-access-r5mtw\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:44.711901 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.711802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdjwf\" (UniqueName: \"kubernetes.io/projected/af08d42d-deb1-41d7-9081-52329edf2111-kube-api-access-qdjwf\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:44.712289 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.712269 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bab5ef01-e412-4447-a555-2913d63a81c6-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:44.720458 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.720409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mtw\" (UniqueName: \"kubernetes.io/projected/bab5ef01-e412-4447-a555-2913d63a81c6-kube-api-access-r5mtw\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:44.812313 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.812280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwvl5\" (UniqueName: \"kubernetes.io/projected/8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84-kube-api-access-qwvl5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2xhfd\" (UID: \"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.812479 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.812337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:44.812479 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.812372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdjwf\" (UniqueName: \"kubernetes.io/projected/af08d42d-deb1-41d7-9081-52329edf2111-kube-api-access-qdjwf\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:44.812479 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.812401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2xhfd\" (UID: \"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.812623 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:44.812576 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 18:17:44.812672 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.812621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2xhfd\" (UID: \"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.812672 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:44.812634 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls podName:af08d42d-deb1-41d7-9081-52329edf2111 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:45.312620489 +0000 UTC m=+133.539296807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lzlc9" (UID: "af08d42d-deb1-41d7-9081-52329edf2111") : secret "samples-operator-tls" not found Sep 11 18:17:44.812989 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.812955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2xhfd\" (UID: \"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.814760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.814742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2xhfd\" (UID: \"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.824682 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.824661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwvl5\" (UniqueName: \"kubernetes.io/projected/8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84-kube-api-access-qwvl5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2xhfd\" (UID: \"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:44.824966 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.824946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdjwf\" (UniqueName: \"kubernetes.io/projected/af08d42d-deb1-41d7-9081-52329edf2111-kube-api-access-qdjwf\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:44.879902 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:44.879870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" Sep 11 18:17:45.144297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:45.144265 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd"] Sep 11 18:17:45.146757 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:17:45.146735 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efe5c71_76f2_4ff3_9793_b8d4dcdb7d84.slice/crio-c5c98a09c8a6c0b8a90807c95d33e1a1a817ebde7db1a925e92c5b1222673bb9 WatchSource:0}: Error finding container c5c98a09c8a6c0b8a90807c95d33e1a1a817ebde7db1a925e92c5b1222673bb9: Status 404 returned error can't find the container with id c5c98a09c8a6c0b8a90807c95d33e1a1a817ebde7db1a925e92c5b1222673bb9 Sep 11 18:17:45.215952 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:45.215912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:45.216119 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:45.216062 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:45.216159 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:45.216124 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls podName:bab5ef01-e412-4447-a555-2913d63a81c6 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:46.216108739 +0000 UTC m=+134.442785074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-jf6v4" (UID: "bab5ef01-e412-4447-a555-2913d63a81c6") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:45.316753 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:45.316696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:45.316886 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:45.316830 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 18:17:45.316945 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:45.316891 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls podName:af08d42d-deb1-41d7-9081-52329edf2111 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:46.31687586 +0000 UTC m=+134.543552177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lzlc9" (UID: "af08d42d-deb1-41d7-9081-52329edf2111") : secret "samples-operator-tls" not found Sep 11 18:17:45.657451 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:45.657413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" event={"ID":"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84","Type":"ContainerStarted","Data":"c5c98a09c8a6c0b8a90807c95d33e1a1a817ebde7db1a925e92c5b1222673bb9"} Sep 11 18:17:46.056042 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:46.056002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:46.221493 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:46.221458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:46.221696 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:46.221594 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:46.221787 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:46.221698 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls podName:bab5ef01-e412-4447-a555-2913d63a81c6 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:48.22168359 +0000 UTC m=+136.448359909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-jf6v4" (UID: "bab5ef01-e412-4447-a555-2913d63a81c6") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:46.322269 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:46.322182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:46.322425 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:46.322347 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 18:17:46.322425 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:46.322424 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls podName:af08d42d-deb1-41d7-9081-52329edf2111 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:48.322402977 +0000 UTC m=+136.549079307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lzlc9" (UID: "af08d42d-deb1-41d7-9081-52329edf2111") : secret "samples-operator-tls" not found Sep 11 18:17:46.516820 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:46.516781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:46.580606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:46.580520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:47.662947 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:47.662864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" event={"ID":"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84","Type":"ContainerStarted","Data":"8f2dc3cd42b3b4ec404d43fce77526c6387e0a916dfa97e4cea98a3cfab7c00d"} Sep 11 18:17:47.679511 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:47.679471 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" podStartSLOduration=1.518719411 podStartE2EDuration="3.6794575s" totalImagesPullingTime="2.160738089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:47.678123309 +0000 UTC m=+135.904799649" watchObservedRunningTime="2026-09-11 18:17:47.6794575 +0000 UTC m=+135.906133840" Sep 11 18:17:48.237433 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:48.237403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:48.237636 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:48.237549 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:48.237636 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:48.237607 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls podName:bab5ef01-e412-4447-a555-2913d63a81c6 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:52.237592723 +0000 UTC m=+140.464269046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-jf6v4" (UID: "bab5ef01-e412-4447-a555-2913d63a81c6") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:48.338143 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:48.338106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:48.338324 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:48.338260 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 18:17:48.338374 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:48.338326 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls podName:af08d42d-deb1-41d7-9081-52329edf2111 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:52.338311008 +0000 UTC m=+140.564987331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lzlc9" (UID: "af08d42d-deb1-41d7-9081-52329edf2111") : secret "samples-operator-tls" not found Sep 11 18:17:48.464276 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:48.464242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:48.559080 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:48.559018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:48.658556 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:48.658525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:48.975171 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:48.975103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:49.067154 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.067118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:49.281533 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.281503 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pjtwp"] Sep 11 18:17:49.283626 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.283611 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.286744 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.286724 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-psjlx\"" Sep 11 18:17:49.286961 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.286943 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 18:17:49.287728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.287699 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 18:17:49.287781 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.287739 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 18:17:49.287824 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.287720 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 18:17:49.295463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.295443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:49.298403 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.298382 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pjtwp"] Sep 11 18:17:49.345129 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.345098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3504eb3-b69b-49e0-a793-2ede1f938615-crio-socket\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.345129 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.345128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3504eb3-b69b-49e0-a793-2ede1f938615-data-volume\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.345308 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.345159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.345308 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.345227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.345308 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.345253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccml6\" (UniqueName: \"kubernetes.io/projected/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-api-access-ccml6\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.370181 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.370151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:49.446037 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.446000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3504eb3-b69b-49e0-a793-2ede1f938615-crio-socket\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.446037 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.446039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3504eb3-b69b-49e0-a793-2ede1f938615-data-volume\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.446283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.446075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.446283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.446100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.446283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.446123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccml6\" (UniqueName: \"kubernetes.io/projected/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-api-access-ccml6\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.446283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.446145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3504eb3-b69b-49e0-a793-2ede1f938615-crio-socket\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.446283 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:49.446199 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:17:49.446283 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:49.446228 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:17:49.446283 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:49.446281 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:49.946264261 +0000 UTC m=+138.172940584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : secret "insights-runtime-extractor-tls" not found Sep 11 18:17:49.446529 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:49.446303 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:49.946296582 +0000 UTC m=+138.172972900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : configmap "kube-rbac-proxy" not found Sep 11 18:17:49.446529 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.446370 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3504eb3-b69b-49e0-a793-2ede1f938615-data-volume\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.454653 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.454623 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccml6\" (UniqueName: \"kubernetes.io/projected/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-api-access-ccml6\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.688557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.688472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:49.787948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.787913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:49.950958 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.950872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.950958 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:49.950909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:49.951165 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:49.950981 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:17:49.951165 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:49.951031 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:50.951017343 +0000 UTC m=+139.177693660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : configmap "kube-rbac-proxy" not found Sep 11 18:17:49.951165 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:49.951033 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:17:49.951165 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:49.951097 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:50.951080128 +0000 UTC m=+139.177756446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : secret "insights-runtime-extractor-tls" not found Sep 11 18:17:50.958720 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:50.958671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:50.958720 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:50.958720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:50.959151 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:50.958833 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:17:50.959151 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:50.958895 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:52.958879795 +0000 UTC m=+141.185556114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : secret "insights-runtime-extractor-tls" not found Sep 11 18:17:50.959151 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:50.958839 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:17:50.959151 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:50.958928 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:52.958921985 +0000 UTC m=+141.185598304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : configmap "kube-rbac-proxy" not found Sep 11 18:17:51.046308 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:51.046276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:51.147318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:51.147287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:51.320791 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:51.320747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:51.391862 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:51.391829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:52.053397 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:52.053370 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mn77k_27a6d135-2518-45ee-b801-709400c2a1a0/dns-node-resolver/0.log" Sep 11 18:17:52.270323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:52.270290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:17:52.270493 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:52.270432 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:52.270493 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:52.270487 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls podName:bab5ef01-e412-4447-a555-2913d63a81c6 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:00.270473458 +0000 UTC m=+148.497149775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-jf6v4" (UID: "bab5ef01-e412-4447-a555-2913d63a81c6") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:17:52.371721 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:52.371625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:17:52.371852 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:52.371792 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 18:17:52.371886 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:52.371852 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls podName:af08d42d-deb1-41d7-9081-52329edf2111 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:00.371835862 +0000 UTC m=+148.598512188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lzlc9" (UID: "af08d42d-deb1-41d7-9081-52329edf2111") : secret "samples-operator-tls" not found Sep 11 18:17:52.608178 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:52.608138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:52.975445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:52.975407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:52.975445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:52.975447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:52.975655 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:52.975550 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:17:52.975655 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:52.975558 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:17:52.975655 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:52.975595 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:56.975582319 +0000 UTC m=+145.202258642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : configmap "kube-rbac-proxy" not found Sep 11 18:17:52.975655 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:52.975609 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:56.975603762 +0000 UTC m=+145.202280079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : secret "insights-runtime-extractor-tls" not found Sep 11 18:17:53.252513 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:53.252479 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkmbt_cbaaf296-872b-4b25-b8da-17a8f77ad1d3/node-ca/0.log" Sep 11 18:17:53.651904 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:53.651826 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4f5s2_0524f5b1-df3e-4288-8864-709be1f0f4e4/serve-healthcheck-canary/0.log" Sep 11 18:17:54.653855 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:54.653824 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-2xhfd_8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84/kube-storage-version-migrator-operator/0.log" Sep 11 18:17:55.206419 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:55.206381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:55.303524 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:55.303484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:55.562343 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:55.562308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:55.678452 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:55.678419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:56.106156 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:56.106118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:56.696718 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:56.696661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:57.005053 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:57.005024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:57.005208 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:57.005060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:17:57.005208 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:57.005144 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:17:57.005208 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:57.005170 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:17:57.005208 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:57.005196 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:05.005182236 +0000 UTC m=+153.231858555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : configmap "kube-rbac-proxy" not found Sep 11 18:17:57.005356 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:17:57.005233 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:05.005219324 +0000 UTC m=+153.231895642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : secret "insights-runtime-extractor-tls" not found Sep 11 18:17:57.184604 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:57.184565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:57.497138 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:57.497054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:57.708566 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:57.708539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:58.316799 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:58.316762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:58.704358 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:58.704257 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:58.929006 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:58.928974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:59.099591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:59.099556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:59.207306 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:59.207272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:59.345487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:59.345454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:59.906238 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:17:59.906207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:00.302616 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:00.302580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:00.337262 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:00.337230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:18:00.337397 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:00.337378 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:18:00.337470 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:00.337459 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls podName:bab5ef01-e412-4447-a555-2913d63a81c6 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:16.337439759 +0000 UTC m=+164.564116076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-jf6v4" (UID: "bab5ef01-e412-4447-a555-2913d63a81c6") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:18:00.438330 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:00.438287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:18:00.440653 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:00.440635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af08d42d-deb1-41d7-9081-52329edf2111-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lzlc9\" (UID: \"af08d42d-deb1-41d7-9081-52329edf2111\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:18:00.484268 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:00.484235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" Sep 11 18:18:00.705723 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:00.705601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:00.751787 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:00.751757 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9"] Sep 11 18:18:00.902473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:00.902431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:01.515384 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:01.515350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:01.614066 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:01.614034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:01.696827 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:01.696792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" event={"ID":"af08d42d-deb1-41d7-9081-52329edf2111","Type":"ContainerStarted","Data":"0715e70c4ed253d21336f2d2ab840d699f90528d51708c81cbab39a09b14dab8"} Sep 11 18:18:01.715116 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:01.715074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:02.111821 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:02.111772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:02.305624 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:02.305594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:02.693511 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:02.693423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:02.701084 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:02.701055 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" event={"ID":"af08d42d-deb1-41d7-9081-52329edf2111","Type":"ContainerStarted","Data":"c5fd20f688e70396c7c0401c3a3bb243bacafb8bfc7f00c76bbaa384aa32ea6f"} Sep 11 18:18:02.701238 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:02.701106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" event={"ID":"af08d42d-deb1-41d7-9081-52329edf2111","Type":"ContainerStarted","Data":"c77f1a3a836a87ad0d297c48eb6c60487797b8294e695749f2f5de975c909563"} Sep 11 18:18:02.719782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:02.719730 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lzlc9" podStartSLOduration=17.243761684 podStartE2EDuration="18.719694527s" totalImagesPullingTime="1.475932843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:02.718866641 +0000 UTC m=+150.945542994" watchObservedRunningTime="2026-09-11 18:18:02.719694527 +0000 UTC m=+150.946370868" Sep 11 18:18:02.901084 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:02.901043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:03.319431 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:03.319393 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:04.101111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:04.101079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:04.436372 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:04.436293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:04.537650 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:04.537616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:04.700910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:04.700833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:04.906259 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:04.906215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:05.080209 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:05.080178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:18:05.080209 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:05.080215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:18:05.080423 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:05.080344 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:18:05.080423 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:05.080421 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls podName:b3504eb3-b69b-49e0-a793-2ede1f938615 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:21.080404952 +0000 UTC m=+169.307081270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pjtwp" (UID: "b3504eb3-b69b-49e0-a793-2ede1f938615") : secret "insights-runtime-extractor-tls" not found Sep 11 18:18:05.080851 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:05.080834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3504eb3-b69b-49e0-a793-2ede1f938615-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:18:05.308830 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:05.308795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:05.703208 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:05.703177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:05.897033 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:05.896993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:06.525367 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:06.525336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:06.587807 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:06.587774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:09.093034 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:09.092989 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" podUID="050a7f57-b326-4310-8617-5075add177fb" Sep 11 18:18:09.101315 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:09.101280 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bhn27" podUID="7c2e0dbc-ad02-49f5-8604-4a45cfba2237" Sep 11 18:18:09.161998 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:09.161969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:09.233505 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:09.233464 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bw29p" podUID="03fbffc5-9471-44da-b5b0-4b881b17898c" Sep 11 18:18:09.262758 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:09.262721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:09.363648 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:09.363561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:09.463072 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:09.463041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:09.677864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:09.677782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:09.724519 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:09.724486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:18:09.724519 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:09.724518 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bhn27" Sep 11 18:18:09.858985 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:09.858955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:09.924888 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:09.924856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:10.239869 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.239837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:10.340386 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.340350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:10.635013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.634980 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf"] Sep 11 18:18:10.639580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.639377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.642318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.642291 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 18:18:10.642446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.642419 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 18:18:10.642492 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.642421 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 18:18:10.643561 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.643539 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 11 18:18:10.653468 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.653443 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf"] Sep 11 18:18:10.723650 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.723611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:10.728332 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.728307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlqcg\" (UniqueName: \"kubernetes.io/projected/372239cf-90cb-4dbe-abf1-07f73c2be5dd-kube-api-access-tlqcg\") pod \"klusterlet-addon-workmgr-5b97cfbcf7-nnzkf\" (UID: \"372239cf-90cb-4dbe-abf1-07f73c2be5dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.728447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.728358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/372239cf-90cb-4dbe-abf1-07f73c2be5dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b97cfbcf7-nnzkf\" (UID: \"372239cf-90cb-4dbe-abf1-07f73c2be5dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.728447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.728392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/372239cf-90cb-4dbe-abf1-07f73c2be5dd-tmp\") pod \"klusterlet-addon-workmgr-5b97cfbcf7-nnzkf\" (UID: \"372239cf-90cb-4dbe-abf1-07f73c2be5dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.730145 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.730125 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-2khhb"] Sep 11 18:18:10.732892 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.732878 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-2khhb" Sep 11 18:18:10.738943 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:10.738910 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-cz7w7\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"default-dockercfg-cz7w7\"" type="*v1.Secret" Sep 11 18:18:10.739051 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:10.738983 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 11 18:18:10.739051 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:10.739000 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 11 18:18:10.741367 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:10.741342 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"downloads-76f99584f4-2khhb\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="394a1856-2042-4876-8df1-7701baa4de0f" pod="openshift-console/downloads-76f99584f4-2khhb" Sep 11 18:18:10.742124 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.742108 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p"] Sep 11 18:18:10.744903 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.744890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" Sep 11 18:18:10.754157 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.754137 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 18:18:10.754260 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.754247 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-p9n6j\"" Sep 11 18:18:10.754408 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.754394 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 18:18:10.772793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.772765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p"] Sep 11 18:18:10.787382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.787350 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-2khhb"] Sep 11 18:18:10.828690 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.828653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d73d283b-2bdf-473f-9833-67b7c2ca8a62-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-bf96p\" (UID: \"d73d283b-2bdf-473f-9833-67b7c2ca8a62\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" Sep 11 18:18:10.828690 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.828692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlqcg\" (UniqueName: \"kubernetes.io/projected/372239cf-90cb-4dbe-abf1-07f73c2be5dd-kube-api-access-tlqcg\") pod \"klusterlet-addon-workmgr-5b97cfbcf7-nnzkf\" (UID: \"372239cf-90cb-4dbe-abf1-07f73c2be5dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.828912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.828736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d73d283b-2bdf-473f-9833-67b7c2ca8a62-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bf96p\" (UID: \"d73d283b-2bdf-473f-9833-67b7c2ca8a62\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" Sep 11 18:18:10.828912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.828890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/372239cf-90cb-4dbe-abf1-07f73c2be5dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b97cfbcf7-nnzkf\" (UID: \"372239cf-90cb-4dbe-abf1-07f73c2be5dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.829033 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.828931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/372239cf-90cb-4dbe-abf1-07f73c2be5dd-tmp\") pod \"klusterlet-addon-workmgr-5b97cfbcf7-nnzkf\" (UID: \"372239cf-90cb-4dbe-abf1-07f73c2be5dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.829385 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.829368 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/372239cf-90cb-4dbe-abf1-07f73c2be5dd-tmp\") pod \"klusterlet-addon-workmgr-5b97cfbcf7-nnzkf\" (UID: \"372239cf-90cb-4dbe-abf1-07f73c2be5dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.831594 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.831569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/372239cf-90cb-4dbe-abf1-07f73c2be5dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b97cfbcf7-nnzkf\" (UID: \"372239cf-90cb-4dbe-abf1-07f73c2be5dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.846065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.846041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlqcg\" (UniqueName: \"kubernetes.io/projected/372239cf-90cb-4dbe-abf1-07f73c2be5dd-kube-api-access-tlqcg\") pod \"klusterlet-addon-workmgr-5b97cfbcf7-nnzkf\" (UID: \"372239cf-90cb-4dbe-abf1-07f73c2be5dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.892607 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.892513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:10.929425 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.929393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d73d283b-2bdf-473f-9833-67b7c2ca8a62-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-bf96p\" (UID: \"d73d283b-2bdf-473f-9833-67b7c2ca8a62\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" Sep 11 18:18:10.929425 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.929430 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45hq\" (UniqueName: \"kubernetes.io/projected/394a1856-2042-4876-8df1-7701baa4de0f-kube-api-access-m45hq\") pod \"downloads-76f99584f4-2khhb\" (UID: \"394a1856-2042-4876-8df1-7701baa4de0f\") " pod="openshift-console/downloads-76f99584f4-2khhb" Sep 11 18:18:10.929618 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.929455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d73d283b-2bdf-473f-9833-67b7c2ca8a62-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bf96p\" (UID: \"d73d283b-2bdf-473f-9833-67b7c2ca8a62\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" Sep 11 18:18:10.930003 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.929981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d73d283b-2bdf-473f-9833-67b7c2ca8a62-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-bf96p\" (UID: \"d73d283b-2bdf-473f-9833-67b7c2ca8a62\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" Sep 11 18:18:10.931757 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.931731 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d73d283b-2bdf-473f-9833-67b7c2ca8a62-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bf96p\" (UID: \"d73d283b-2bdf-473f-9833-67b7c2ca8a62\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" Sep 11 18:18:10.948567 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.948547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:10.981301 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:10.981265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:11.029765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.029726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45hq\" (UniqueName: \"kubernetes.io/projected/394a1856-2042-4876-8df1-7701baa4de0f-kube-api-access-m45hq\") pod \"downloads-76f99584f4-2khhb\" (UID: \"394a1856-2042-4876-8df1-7701baa4de0f\") " pod="openshift-console/downloads-76f99584f4-2khhb" Sep 11 18:18:11.052152 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.052126 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" Sep 11 18:18:11.083800 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.083768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:11.219258 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.219229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf"] Sep 11 18:18:11.219365 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:11.219294 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372239cf_90cb_4dbe_abf1_07f73c2be5dd.slice/crio-a7830b7d7401e0ee1c2ff0ab30299d4f51d0666d2aaa28684a3b72ee00ebf31e WatchSource:0}: Error finding container a7830b7d7401e0ee1c2ff0ab30299d4f51d0666d2aaa28684a3b72ee00ebf31e: Status 404 returned error can't find the container with id a7830b7d7401e0ee1c2ff0ab30299d4f51d0666d2aaa28684a3b72ee00ebf31e Sep 11 18:18:11.317344 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.317314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p"] Sep 11 18:18:11.320527 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:11.320499 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73d283b_2bdf_473f_9833_67b7c2ca8a62.slice/crio-98f1b5b9c14fddd5900cefab669126d8207b118bc063c21b38d930e4627fdcb7 WatchSource:0}: Error finding container 98f1b5b9c14fddd5900cefab669126d8207b118bc063c21b38d930e4627fdcb7: Status 404 returned error can't find the container with id 98f1b5b9c14fddd5900cefab669126d8207b118bc063c21b38d930e4627fdcb7 Sep 11 18:18:11.583087 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.583058 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-cz7w7\"" Sep 11 18:18:11.731404 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.731367 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" event={"ID":"372239cf-90cb-4dbe-abf1-07f73c2be5dd","Type":"ContainerStarted","Data":"a7830b7d7401e0ee1c2ff0ab30299d4f51d0666d2aaa28684a3b72ee00ebf31e"} Sep 11 18:18:11.732436 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.732403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" event={"ID":"d73d283b-2bdf-473f-9833-67b7c2ca8a62","Type":"ContainerStarted","Data":"98f1b5b9c14fddd5900cefab669126d8207b118bc063c21b38d930e4627fdcb7"} Sep 11 18:18:11.875866 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.875785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:11.920863 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.920797 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 18:18:11.977761 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:11.977722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:12.114784 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:12.114756 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 18:18:12.122164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:12.122124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45hq\" (UniqueName: \"kubernetes.io/projected/394a1856-2042-4876-8df1-7701baa4de0f-kube-api-access-m45hq\") pod \"downloads-76f99584f4-2khhb\" (UID: \"394a1856-2042-4876-8df1-7701baa4de0f\") " pod="openshift-console/downloads-76f99584f4-2khhb" Sep 11 18:18:12.241009 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:12.240936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-2khhb" Sep 11 18:18:12.652059 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:12.652027 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-2khhb"] Sep 11 18:18:12.654773 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:12.654752 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394a1856_2042_4876_8df1_7701baa4de0f.slice/crio-0d97df1c8fc456502da7762f5563638254d39c400fc45a1b2fbea3f9d385f4dc WatchSource:0}: Error finding container 0d97df1c8fc456502da7762f5563638254d39c400fc45a1b2fbea3f9d385f4dc: Status 404 returned error can't find the container with id 0d97df1c8fc456502da7762f5563638254d39c400fc45a1b2fbea3f9d385f4dc Sep 11 18:18:12.736487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:12.736449 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" event={"ID":"d73d283b-2bdf-473f-9833-67b7c2ca8a62","Type":"ContainerStarted","Data":"be4ef2328a5b7aab2181982cadf395e37475ae1563f0cda1eed960229c150402"} Sep 11 18:18:12.737642 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:12.737619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-2khhb" event={"ID":"394a1856-2042-4876-8df1-7701baa4de0f","Type":"ContainerStarted","Data":"0d97df1c8fc456502da7762f5563638254d39c400fc45a1b2fbea3f9d385f4dc"} Sep 11 18:18:12.753196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:12.753151 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bf96p" podStartSLOduration=1.6390068119999999 podStartE2EDuration="2.753135942s" totalImagesPullingTime="1.11412913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:12.751594647 +0000 UTC m=+160.978270999" watchObservedRunningTime="2026-09-11 18:18:12.753135942 +0000 UTC m=+160.979812316" Sep 11 18:18:12.860429 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:12.860397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:12.962164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:12.962131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:13.164605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:13.164503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:14.052343 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.052299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:18:14.052813 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.052454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:18:14.054920 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.054895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c2e0dbc-ad02-49f5-8604-4a45cfba2237-metrics-tls\") pod \"dns-default-bhn27\" (UID: \"7c2e0dbc-ad02-49f5-8604-4a45cfba2237\") " pod="openshift-dns/dns-default-bhn27" Sep 11 18:18:14.055058 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.054968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"image-registry-648d6bd57b-sljfs\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:18:14.228377 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.228338 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksdw9\"" Sep 11 18:18:14.228377 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.228340 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-grmtt\"" Sep 11 18:18:14.235668 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.235638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bhn27" Sep 11 18:18:14.235841 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.235734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:18:14.523407 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.523381 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bhn27"] Sep 11 18:18:14.526189 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.526157 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648d6bd57b-sljfs"] Sep 11 18:18:14.526807 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:14.526770 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2e0dbc_ad02_49f5_8604_4a45cfba2237.slice/crio-cc0061d3ac8957a7f028af4f41f610feb46333350c1482cf04cdd181df2b48c5 WatchSource:0}: Error finding container cc0061d3ac8957a7f028af4f41f610feb46333350c1482cf04cdd181df2b48c5: Status 404 returned error can't find the container with id cc0061d3ac8957a7f028af4f41f610feb46333350c1482cf04cdd181df2b48c5 Sep 11 18:18:14.529371 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:14.529348 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050a7f57_b326_4310_8617_5075add177fb.slice/crio-f161267727dbe4e3d876fa08c8e2ea50eb2626ba6017a2a86eab2f109f6cc941 WatchSource:0}: Error finding container f161267727dbe4e3d876fa08c8e2ea50eb2626ba6017a2a86eab2f109f6cc941: Status 404 returned error can't find the container with id f161267727dbe4e3d876fa08c8e2ea50eb2626ba6017a2a86eab2f109f6cc941 Sep 11 18:18:14.750122 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.750082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" event={"ID":"050a7f57-b326-4310-8617-5075add177fb","Type":"ContainerStarted","Data":"192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e"} Sep 11 18:18:14.750394 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.750145 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" event={"ID":"050a7f57-b326-4310-8617-5075add177fb","Type":"ContainerStarted","Data":"f161267727dbe4e3d876fa08c8e2ea50eb2626ba6017a2a86eab2f109f6cc941"} Sep 11 18:18:14.750394 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.750215 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:18:14.751823 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.751793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bhn27" event={"ID":"7c2e0dbc-ad02-49f5-8604-4a45cfba2237","Type":"ContainerStarted","Data":"cc0061d3ac8957a7f028af4f41f610feb46333350c1482cf04cdd181df2b48c5"} Sep 11 18:18:14.771499 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:14.771274 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" podStartSLOduration=162.771253011 podStartE2EDuration="2m42.771253011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:14.770788205 +0000 UTC m=+162.997464567" watchObservedRunningTime="2026-09-11 18:18:14.771253011 +0000 UTC m=+162.997929352" Sep 11 18:18:16.375579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:16.375534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:18:16.379950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:16.379877 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bab5ef01-e412-4447-a555-2913d63a81c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-jf6v4\" (UID: \"bab5ef01-e412-4447-a555-2913d63a81c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:18:16.581190 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:16.581146 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" Sep 11 18:18:18.277269 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:18.277213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:18.341331 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:18.341289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:18.984303 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:18.984223 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4"] Sep 11 18:18:18.987320 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:18.987288 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab5ef01_e412_4447_a555_2913d63a81c6.slice/crio-6ea45fb242ce30639bed528daad88c1b9aa6f841499f015e0acd6b58a2ff3327 WatchSource:0}: Error finding container 6ea45fb242ce30639bed528daad88c1b9aa6f841499f015e0acd6b58a2ff3327: Status 404 returned error can't find the container with id 6ea45fb242ce30639bed528daad88c1b9aa6f841499f015e0acd6b58a2ff3327 Sep 11 18:18:19.777338 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:19.777295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" event={"ID":"bab5ef01-e412-4447-a555-2913d63a81c6","Type":"ContainerStarted","Data":"6ea45fb242ce30639bed528daad88c1b9aa6f841499f015e0acd6b58a2ff3327"} Sep 11 18:18:19.778940 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:19.778911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" event={"ID":"372239cf-90cb-4dbe-abf1-07f73c2be5dd","Type":"ContainerStarted","Data":"30359a77b3eb6a02fdfe78b5883ce2d5e0a8f00d1a1488e5254c6b87ecfc9ea8"} Sep 11 18:18:19.779185 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:19.779130 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:19.781111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:19.781085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bhn27" event={"ID":"7c2e0dbc-ad02-49f5-8604-4a45cfba2237","Type":"ContainerStarted","Data":"6646c3b02e2c17c834b884005a2537ccc945e789a7a1b37ea5afafc5e5645d60"} Sep 11 18:18:19.781221 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:19.781132 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bhn27" event={"ID":"7c2e0dbc-ad02-49f5-8604-4a45cfba2237","Type":"ContainerStarted","Data":"d6f00f09826e42a984dd1584ed6b48368abae8d3a4e461734eed83757adabc10"} Sep 11 18:18:19.781326 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:19.781311 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" Sep 11 18:18:19.781387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:19.781344 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bhn27" Sep 11 18:18:19.795351 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:19.795302 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b97cfbcf7-nnzkf" podStartSLOduration=2.292653343 podStartE2EDuration="9.795285562s" totalImagesPullingTime="7.502632219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:19.794487926 +0000 UTC m=+168.021164292" watchObservedRunningTime="2026-09-11 18:18:19.795285562 +0000 UTC m=+168.021961905" Sep 11 18:18:19.826379 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:19.826327 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bhn27" podStartSLOduration=129.6486996 podStartE2EDuration="2m13.826308275s" totalImagesPullingTime="4.177608675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:19.825918779 +0000 UTC m=+168.052595148" watchObservedRunningTime="2026-09-11 18:18:19.826308275 +0000 UTC m=+168.052984595" Sep 11 18:18:20.268606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:20.268567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:20.320997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:20.320962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:21.117592 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.117548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:18:21.120543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.120511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3504eb3-b69b-49e0-a793-2ede1f938615-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjtwp\" (UID: \"b3504eb3-b69b-49e0-a793-2ede1f938615\") " pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:18:21.222722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.222676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:18:21.391775 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.391683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pjtwp" Sep 11 18:18:21.722321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.722095 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pjtwp"] Sep 11 18:18:21.725201 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:21.725174 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3504eb3_b69b_49e0_a793_2ede1f938615.slice/crio-f9908f67b60faa6b73d05c2b31c993cda072cf78c167706029d61b430a8e84b4 WatchSource:0}: Error finding container f9908f67b60faa6b73d05c2b31c993cda072cf78c167706029d61b430a8e84b4: Status 404 returned error can't find the container with id f9908f67b60faa6b73d05c2b31c993cda072cf78c167706029d61b430a8e84b4 Sep 11 18:18:21.790523 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.790473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjtwp" event={"ID":"b3504eb3-b69b-49e0-a793-2ede1f938615","Type":"ContainerStarted","Data":"f9908f67b60faa6b73d05c2b31c993cda072cf78c167706029d61b430a8e84b4"} Sep 11 18:18:21.792425 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.792395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" event={"ID":"bab5ef01-e412-4447-a555-2913d63a81c6","Type":"ContainerStarted","Data":"bede820ab1716685063b29084091927e69d65382063b2f6aea683f4f698b1b64"} Sep 11 18:18:21.817832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.817766 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-jf6v4" podStartSLOduration=35.316597303 podStartE2EDuration="37.817745491s" totalImagesPullingTime="2.501148188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:21.816592495 +0000 UTC m=+170.043268856" watchObservedRunningTime="2026-09-11 18:18:21.817745491 +0000 UTC m=+170.044421832" Sep 11 18:18:21.841745 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.841694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:21.940248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:21.940211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:22.004718 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.004669 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg"] Sep 11 18:18:22.007961 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.007938 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" Sep 11 18:18:22.010460 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.010441 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-75kn8\"" Sep 11 18:18:22.010629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.010456 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 18:18:22.016978 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.016955 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg"] Sep 11 18:18:22.025325 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.025301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:22.126889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.126859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/37608dbc-bbcf-484e-927a-457ccafcf4c8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-6svkg\" (UID: \"37608dbc-bbcf-484e-927a-457ccafcf4c8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" Sep 11 18:18:22.227721 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.227672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/37608dbc-bbcf-484e-927a-457ccafcf4c8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-6svkg\" (UID: \"37608dbc-bbcf-484e-927a-457ccafcf4c8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" Sep 11 18:18:22.227906 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:22.227817 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 11 18:18:22.227906 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:22.227879 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37608dbc-bbcf-484e-927a-457ccafcf4c8-tls-certificates podName:37608dbc-bbcf-484e-927a-457ccafcf4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:22.727859405 +0000 UTC m=+170.954535726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/37608dbc-bbcf-484e-927a-457ccafcf4c8-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-6svkg" (UID: "37608dbc-bbcf-484e-927a-457ccafcf4c8") : secret "prometheus-operator-admission-webhook-tls" not found Sep 11 18:18:22.359555 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.359509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:22.731202 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.731117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/37608dbc-bbcf-484e-927a-457ccafcf4c8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-6svkg\" (UID: \"37608dbc-bbcf-484e-927a-457ccafcf4c8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" Sep 11 18:18:22.733795 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.733767 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/37608dbc-bbcf-484e-927a-457ccafcf4c8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-6svkg\" (UID: \"37608dbc-bbcf-484e-927a-457ccafcf4c8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" Sep 11 18:18:22.800801 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.800761 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjtwp" event={"ID":"b3504eb3-b69b-49e0-a793-2ede1f938615","Type":"ContainerStarted","Data":"d34617ce6212526fef73d0b530bac7737cf35594ebb1711913f3de0248391e98"} Sep 11 18:18:22.800982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.800864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjtwp" event={"ID":"b3504eb3-b69b-49e0-a793-2ede1f938615","Type":"ContainerStarted","Data":"51432a4a0fb5bc090561319c825db72087078264f8c17a69ddabce8b1e38c9b4"} Sep 11 18:18:22.919726 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:22.919579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" Sep 11 18:18:23.207269 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:23.207195 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg"] Sep 11 18:18:23.210018 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:23.209987 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37608dbc_bbcf_484e_927a_457ccafcf4c8.slice/crio-7a7a873153d81c6aa8c191cb6432dcde328a419d9139020cc0b59f1224c8f446 WatchSource:0}: Error finding container 7a7a873153d81c6aa8c191cb6432dcde328a419d9139020cc0b59f1224c8f446: Status 404 returned error can't find the container with id 7a7a873153d81c6aa8c191cb6432dcde328a419d9139020cc0b59f1224c8f446 Sep 11 18:18:23.805619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:23.805552 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" event={"ID":"37608dbc-bbcf-484e-927a-457ccafcf4c8","Type":"ContainerStarted","Data":"7a7a873153d81c6aa8c191cb6432dcde328a419d9139020cc0b59f1224c8f446"} Sep 11 18:18:24.276870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.276834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:24.336164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.336127 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:24.495743 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.495644 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5998f5dcd7-6nt5h"] Sep 11 18:18:24.498891 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.498868 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.501699 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.501674 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 18:18:24.502616 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.502595 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 18:18:24.502741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.502595 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 18:18:24.502991 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.502964 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 18:18:24.502991 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.502978 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jnzqn\"" Sep 11 18:18:24.503126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.503084 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 18:18:24.510514 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.510493 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5998f5dcd7-6nt5h"] Sep 11 18:18:24.548465 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.548391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-oauth-config\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.548465 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.548433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-service-ca\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.548649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.548552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-serving-cert\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.548649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.548585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-config\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.548649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.548610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmnzb\" (UniqueName: \"kubernetes.io/projected/75f0cb41-f587-46e2-9b3e-609ea22c5d09-kube-api-access-lmnzb\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.548803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.548663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-oauth-serving-cert\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.649161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.649124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-oauth-config\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.649161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.649167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-service-ca\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.649400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.649217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-serving-cert\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.649400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.649232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-config\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.649400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.649251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmnzb\" (UniqueName: \"kubernetes.io/projected/75f0cb41-f587-46e2-9b3e-609ea22c5d09-kube-api-access-lmnzb\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.649400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.649295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-oauth-serving-cert\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.650092 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.650062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-oauth-serving-cert\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.650092 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.650080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-config\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.650268 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.650071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-service-ca\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.652098 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.652075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-serving-cert\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.652332 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.652314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-oauth-config\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.658219 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.658173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmnzb\" (UniqueName: \"kubernetes.io/projected/75f0cb41-f587-46e2-9b3e-609ea22c5d09-kube-api-access-lmnzb\") pod \"console-5998f5dcd7-6nt5h\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:24.810325 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:24.810225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:26.078973 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.078940 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f6bf575fb-ngfzn"] Sep 11 18:18:26.082072 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.082045 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.086447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.086411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:26.091421 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.091395 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 18:18:26.092008 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.091973 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f6bf575fb-ngfzn"] Sep 11 18:18:26.147504 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.147468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:26.164100 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.164064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpwnt\" (UniqueName: \"kubernetes.io/projected/37b49af5-7b8e-4179-86b0-43209bf6db7f-kube-api-access-hpwnt\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.164261 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.164119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-oauth-serving-cert\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.164261 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.164206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-trusted-ca-bundle\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.164261 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.164248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-config\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.164413 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.164279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-serving-cert\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.164413 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.164310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-oauth-config\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.164413 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.164339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-service-ca\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.264953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.264915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-oauth-serving-cert\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.265125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.264972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-trusted-ca-bundle\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.265125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.265006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-config\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.265125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.265034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-serving-cert\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.265125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.265062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-oauth-config\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.265125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.265089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-service-ca\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.265400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.265272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpwnt\" (UniqueName: \"kubernetes.io/projected/37b49af5-7b8e-4179-86b0-43209bf6db7f-kube-api-access-hpwnt\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.266052 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.265796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-service-ca\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.266052 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.265832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-config\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.266294 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.266272 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-oauth-serving-cert\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.266433 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.266408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-trusted-ca-bundle\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.267755 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.267734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-serving-cert\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.267974 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.267954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-oauth-config\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.273738 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.273684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpwnt\" (UniqueName: \"kubernetes.io/projected/37b49af5-7b8e-4179-86b0-43209bf6db7f-kube-api-access-hpwnt\") pod \"console-f6bf575fb-ngfzn\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:26.277917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.277887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:26.334449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.334361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:26.394370 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:26.394332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:29.475743 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:29.475691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:29.548926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:29.548871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:29.787248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:29.787219 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bhn27" Sep 11 18:18:31.229768 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.229738 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f6bf575fb-ngfzn"] Sep 11 18:18:31.233005 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.232977 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5998f5dcd7-6nt5h"] Sep 11 18:18:31.233163 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:31.233141 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b49af5_7b8e_4179_86b0_43209bf6db7f.slice/crio-854d214dbbf9946f0c346124d5060fd41ae877e84093a8c6279c1406dacd3c4a WatchSource:0}: Error finding container 854d214dbbf9946f0c346124d5060fd41ae877e84093a8c6279c1406dacd3c4a: Status 404 returned error can't find the container with id 854d214dbbf9946f0c346124d5060fd41ae877e84093a8c6279c1406dacd3c4a Sep 11 18:18:31.236174 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:31.236150 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f0cb41_f587_46e2_9b3e_609ea22c5d09.slice/crio-daf4f093dffa6d7d5a2eaaf44e90a606c661dfef4ee3d5727b1a59011d2a45e7 WatchSource:0}: Error finding container daf4f093dffa6d7d5a2eaaf44e90a606c661dfef4ee3d5727b1a59011d2a45e7: Status 404 returned error can't find the container with id daf4f093dffa6d7d5a2eaaf44e90a606c661dfef4ee3d5727b1a59011d2a45e7 Sep 11 18:18:31.841328 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.841219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6bf575fb-ngfzn" event={"ID":"37b49af5-7b8e-4179-86b0-43209bf6db7f","Type":"ContainerStarted","Data":"854d214dbbf9946f0c346124d5060fd41ae877e84093a8c6279c1406dacd3c4a"} Sep 11 18:18:31.843771 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.843734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-2khhb" event={"ID":"394a1856-2042-4876-8df1-7701baa4de0f","Type":"ContainerStarted","Data":"efd053df721b0fb6163914a0e11e2d61ea8a010d4fda8c7ec86dc67fee19e5cd"} Sep 11 18:18:31.844137 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.843955 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-2khhb" Sep 11 18:18:31.848119 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.848089 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjtwp" event={"ID":"b3504eb3-b69b-49e0-a793-2ede1f938615","Type":"ContainerStarted","Data":"5d7b7d984f82a4cbc77226456144723863e01019ed0185c50aa630f69200c66f"} Sep 11 18:18:31.850539 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.850513 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" event={"ID":"37608dbc-bbcf-484e-927a-457ccafcf4c8","Type":"ContainerStarted","Data":"e9e52cd055d398fbf1696bc0d0de83f4a0565e93ab8d64234fd225aa25adf013"} Sep 11 18:18:31.850967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.850775 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" Sep 11 18:18:31.852820 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.852545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5998f5dcd7-6nt5h" event={"ID":"75f0cb41-f587-46e2-9b3e-609ea22c5d09","Type":"ContainerStarted","Data":"daf4f093dffa6d7d5a2eaaf44e90a606c661dfef4ee3d5727b1a59011d2a45e7"} Sep 11 18:18:31.857483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.857358 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-2khhb" Sep 11 18:18:31.857736 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.857658 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" Sep 11 18:18:31.865142 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.865086 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-2khhb" podStartSLOduration=3.500540725 podStartE2EDuration="21.865066523s" totalImagesPullingTime="18.364525798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:31.862970156 +0000 UTC m=+180.089646519" watchObservedRunningTime="2026-09-11 18:18:31.865066523 +0000 UTC m=+180.091742864" Sep 11 18:18:31.881675 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.881271 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pjtwp" podStartSLOduration=33.802778298 podStartE2EDuration="42.881216835s" totalImagesPullingTime="9.078438537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:17:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:31.879992606 +0000 UTC m=+180.106668969" watchObservedRunningTime="2026-09-11 18:18:31.881216835 +0000 UTC m=+180.107893177" Sep 11 18:18:31.897884 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:31.897826 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-6svkg" podStartSLOduration=3.094547006 podStartE2EDuration="10.897808934s" totalImagesPullingTime="7.803261928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:31.897720351 +0000 UTC m=+180.124396683" watchObservedRunningTime="2026-09-11 18:18:31.897808934 +0000 UTC m=+180.124485276" Sep 11 18:18:32.032978 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:32.032818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:32.107096 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:32.106899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:32.179068 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:32.178948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:33.075691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:33.075604 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648d6bd57b-sljfs"] Sep 11 18:18:33.081682 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:33.081548 2579 patch_prober.go:28] interesting pod/image-registry-648d6bd57b-sljfs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 18:18:33.081682 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:33.081615 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" podUID="050a7f57-b326-4310-8617-5075add177fb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:18:33.253419 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:33.253382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:33.369918 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:33.369729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:34.124037 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:34.123990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:35.487000 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:35.486955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:35.546117 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:35.546075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:35.872119 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:35.872081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6bf575fb-ngfzn" event={"ID":"37b49af5-7b8e-4179-86b0-43209bf6db7f","Type":"ContainerStarted","Data":"d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450"} Sep 11 18:18:35.873974 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:35.873938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5998f5dcd7-6nt5h" event={"ID":"75f0cb41-f587-46e2-9b3e-609ea22c5d09","Type":"ContainerStarted","Data":"25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36"} Sep 11 18:18:35.890308 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:35.890248 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f6bf575fb-ngfzn" podStartSLOduration=5.547443168 podStartE2EDuration="9.890229301s" totalImagesPullingTime="4.342786133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:35.889596258 +0000 UTC m=+184.116272623" watchObservedRunningTime="2026-09-11 18:18:35.890229301 +0000 UTC m=+184.116905644" Sep 11 18:18:35.909793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:35.909742 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5998f5dcd7-6nt5h" podStartSLOduration=7.562862964 podStartE2EDuration="11.909719214s" totalImagesPullingTime="4.34685625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:35.909057771 +0000 UTC m=+184.135734137" watchObservedRunningTime="2026-09-11 18:18:35.909719214 +0000 UTC m=+184.136395548" Sep 11 18:18:36.212151 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.212064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:36.395073 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.395017 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:36.395285 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.395100 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:36.400694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.400661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:36.464011 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.463917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:36.544838 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.544803 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gj24r"] Sep 11 18:18:36.571258 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.571223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:36.581654 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.581627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.584401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.584374 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 18:18:36.584543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.584488 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kzmdz\"" Sep 11 18:18:36.584696 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.584653 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 18:18:36.584833 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.584752 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 18:18:36.584833 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.584764 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 18:18:36.670760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.670727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0feb5979-7e5c-4f24-8446-34510ef61780-sys\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.670944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.670790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-accelerators-collector-config\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.670944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.670835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-tls\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.670944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.670859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.670944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.670916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0feb5979-7e5c-4f24-8446-34510ef61780-root\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.670944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.670939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0feb5979-7e5c-4f24-8446-34510ef61780-metrics-client-ca\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.671175 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.670972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-textfile\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.671175 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.670992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-wtmp\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.671175 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.671014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75nf\" (UniqueName: \"kubernetes.io/projected/0feb5979-7e5c-4f24-8446-34510ef61780-kube-api-access-d75nf\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.719758 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.719651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:36.772204 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0feb5979-7e5c-4f24-8446-34510ef61780-sys\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772380 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-accelerators-collector-config\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772380 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-tls\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772380 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0feb5979-7e5c-4f24-8446-34510ef61780-sys\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772380 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0feb5979-7e5c-4f24-8446-34510ef61780-root\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772605 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:36.772416 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 18:18:36.772605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0feb5979-7e5c-4f24-8446-34510ef61780-metrics-client-ca\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-textfile\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772605 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:36.772488 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-tls podName:0feb5979-7e5c-4f24-8446-34510ef61780 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:37.272465043 +0000 UTC m=+185.499141379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-tls") pod "node-exporter-gj24r" (UID: "0feb5979-7e5c-4f24-8446-34510ef61780") : secret "node-exporter-tls" not found Sep 11 18:18:36.772605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-wtmp\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75nf\" (UniqueName: \"kubernetes.io/projected/0feb5979-7e5c-4f24-8446-34510ef61780-kube-api-access-d75nf\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0feb5979-7e5c-4f24-8446-34510ef61780-root\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-wtmp\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.772982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.772973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-accelerators-collector-config\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.773155 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.773133 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-textfile\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.773155 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.773146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0feb5979-7e5c-4f24-8446-34510ef61780-metrics-client-ca\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.775000 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.774972 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.783215 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.783185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75nf\" (UniqueName: \"kubernetes.io/projected/0feb5979-7e5c-4f24-8446-34510ef61780-kube-api-access-d75nf\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:36.821678 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.821631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:36.882126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.882096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:18:36.905895 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.905856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:36.931505 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.931475 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5998f5dcd7-6nt5h"] Sep 11 18:18:36.972851 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:36.972762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:37.089878 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.089834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:37.265971 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.265933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:37.278795 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.278758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-tls\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:37.281437 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.281412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0feb5979-7e5c-4f24-8446-34510ef61780-node-exporter-tls\") pod \"node-exporter-gj24r\" (UID: \"0feb5979-7e5c-4f24-8446-34510ef61780\") " pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:37.434141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.434102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:37.492223 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.492185 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gj24r" Sep 11 18:18:37.516068 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.515986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:37.522050 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.522017 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 18:18:37.536623 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:37.536203 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0feb5979_7e5c_4f24_8446_34510ef61780.slice/crio-980b4485e6139455a0eee285a60e74e1661beac7fdc66782be0dac5082536735 WatchSource:0}: Error finding container 980b4485e6139455a0eee285a60e74e1661beac7fdc66782be0dac5082536735: Status 404 returned error can't find the container with id 980b4485e6139455a0eee285a60e74e1661beac7fdc66782be0dac5082536735 Sep 11 18:18:37.572992 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.572935 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 18:18:37.573423 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.573124 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.575786 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.575666 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 18:18:37.576597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.576067 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 18:18:37.576597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.576159 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dslx4\"" Sep 11 18:18:37.576597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.576290 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 18:18:37.576597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.576295 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 18:18:37.576597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.576432 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 18:18:37.576597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.576452 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 18:18:37.576597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.576494 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 18:18:37.576994 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.576656 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 18:18:37.576994 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.576903 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 18:18:37.600810 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.600764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:37.682752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.682690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-web-config\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.682752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.682763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.682817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-volume\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.682886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.682911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.682935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-out\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.682958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.683079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.683148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.683188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.683215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.683239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfnv\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-kube-api-access-trfnv\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.683323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.683270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfnv\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-kube-api-access-trfnv\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-web-config\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-volume\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.784639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.785045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-out\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.785045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.785045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.784697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.785486 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.785455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.788002 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.787519 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.788002 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.787958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.788002 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.787986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.788831 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.788805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.789075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.789052 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.789421 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.789386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-volume\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.790535 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.790260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-out\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.790535 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.790375 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.790843 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.790794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-web-config\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.791304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.791221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.793350 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.793323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfnv\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-kube-api-access-trfnv\") pod \"alertmanager-main-0\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:37.883915 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.883872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gj24r" event={"ID":"0feb5979-7e5c-4f24-8446-34510ef61780","Type":"ContainerStarted","Data":"980b4485e6139455a0eee285a60e74e1661beac7fdc66782be0dac5082536735"} Sep 11 18:18:37.885496 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:37.885471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:18:38.210446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:38.210284 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 18:18:38.214081 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:38.214017 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6383cc78_8370_4ab6_a91c_938bb4ec690f.slice/crio-98ff1f3de8bfee2884276940dc0003b356c931ee84ecb694731372aabb0c77b4 WatchSource:0}: Error finding container 98ff1f3de8bfee2884276940dc0003b356c931ee84ecb694731372aabb0c77b4: Status 404 returned error can't find the container with id 98ff1f3de8bfee2884276940dc0003b356c931ee84ecb694731372aabb0c77b4 Sep 11 18:18:38.294724 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:38.294659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:38.394654 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:38.394612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:38.694903 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:38.694869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:38.889186 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:38.889147 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerStarted","Data":"98ff1f3de8bfee2884276940dc0003b356c931ee84ecb694731372aabb0c77b4"} Sep 11 18:18:38.921307 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:38.921276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:39.040239 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:39.040193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:39.155612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:39.155560 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:39.252488 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:39.252459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:39.444483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:39.444385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:39.548299 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:39.548264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:39.846678 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:39.846635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:39.895276 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:39.895238 2579 generic.go:356] "Generic (PLEG): container finished" podID="0feb5979-7e5c-4f24-8446-34510ef61780" containerID="8cd6e22969ad632ab78081e206ba2abc6d00fdae4e4aa3541845fe01cd6444ec" exitCode=0 Sep 11 18:18:39.895448 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:39.895299 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gj24r" event={"ID":"0feb5979-7e5c-4f24-8446-34510ef61780","Type":"ContainerDied","Data":"8cd6e22969ad632ab78081e206ba2abc6d00fdae4e4aa3541845fe01cd6444ec"} Sep 11 18:18:40.901218 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:40.901178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gj24r" event={"ID":"0feb5979-7e5c-4f24-8446-34510ef61780","Type":"ContainerStarted","Data":"0fa9124b55ff0f2abcee971d5aa65593526054cc41202f43e0e89e9982ccf6c6"} Sep 11 18:18:40.901635 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:40.901240 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gj24r" event={"ID":"0feb5979-7e5c-4f24-8446-34510ef61780","Type":"ContainerStarted","Data":"d769175c74322706c9e04a98acaa92e775b1cc5fe44682f23749243792836bd2"} Sep 11 18:18:40.902834 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:40.902805 2579 generic.go:356] "Generic (PLEG): container finished" podID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerID="d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4" exitCode=0 Sep 11 18:18:40.902953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:40.902877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerDied","Data":"d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4"} Sep 11 18:18:40.923122 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:40.923076 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gj24r" podStartSLOduration=3.252981476 podStartE2EDuration="4.923061633s" totalImagesPullingTime="1.565519223s" totalInitContainerRuntime="104.560934ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:40.921444917 +0000 UTC m=+189.148121277" watchObservedRunningTime="2026-09-11 18:18:40.923061633 +0000 UTC m=+189.149738018" Sep 11 18:18:41.244315 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:41.244226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:41.347172 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:41.347140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:41.442198 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:41.442161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:42.291677 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.291645 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fbd7656bf-jrw2n"] Sep 11 18:18:42.313574 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.313542 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbd7656bf-jrw2n"] Sep 11 18:18:42.313775 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.313650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.429845 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.429804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-oauth-serving-cert\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.430013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.429862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-config\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.430013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.429923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxvz\" (UniqueName: \"kubernetes.io/projected/9b0b5cae-637c-402c-ab3c-09f0c8e94387-kube-api-access-sqxvz\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.430013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.429970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-service-ca\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.430013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.429989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-serving-cert\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.430146 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.430014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-trusted-ca-bundle\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.430146 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.430077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-oauth-config\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.531339 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.531302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-config\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.531339 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.531345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxvz\" (UniqueName: \"kubernetes.io/projected/9b0b5cae-637c-402c-ab3c-09f0c8e94387-kube-api-access-sqxvz\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.531619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.531385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-service-ca\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.531619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.531407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-serving-cert\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.531619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.531430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-trusted-ca-bundle\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.531619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.531459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-oauth-config\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.531619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.531522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-oauth-serving-cert\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.532169 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.532112 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-config\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.532169 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.532113 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-service-ca\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.532307 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.532251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-oauth-serving-cert\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.532575 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.532546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-trusted-ca-bundle\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.533821 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.533796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-oauth-config\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.534111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.534091 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-serving-cert\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.539196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.539178 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxvz\" (UniqueName: \"kubernetes.io/projected/9b0b5cae-637c-402c-ab3c-09f0c8e94387-kube-api-access-sqxvz\") pod \"console-6fbd7656bf-jrw2n\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.591690 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.591619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:42.623862 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.623832 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:42.731379 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.731347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:42.913234 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:42.913204 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerStarted","Data":"a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57"} Sep 11 18:18:43.058144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:43.058108 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbd7656bf-jrw2n"] Sep 11 18:18:43.060685 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:18:43.060660 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0b5cae_637c_402c_ab3c_09f0c8e94387.slice/crio-384da626590cb504b3e9f43548d18c618bae79bf68819edbfc730c0bf3ff3d1f WatchSource:0}: Error finding container 384da626590cb504b3e9f43548d18c618bae79bf68819edbfc730c0bf3ff3d1f: Status 404 returned error can't find the container with id 384da626590cb504b3e9f43548d18c618bae79bf68819edbfc730c0bf3ff3d1f Sep 11 18:18:43.080230 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:43.080205 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:18:43.922215 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:43.922178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerStarted","Data":"a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f"} Sep 11 18:18:43.922665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:43.922255 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerStarted","Data":"f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa"} Sep 11 18:18:43.922665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:43.922297 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerStarted","Data":"ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a"} Sep 11 18:18:43.922665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:43.922326 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerStarted","Data":"30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b"} Sep 11 18:18:43.923670 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:43.923652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbd7656bf-jrw2n" event={"ID":"9b0b5cae-637c-402c-ab3c-09f0c8e94387","Type":"ContainerStarted","Data":"729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61"} Sep 11 18:18:43.923727 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:43.923687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbd7656bf-jrw2n" event={"ID":"9b0b5cae-637c-402c-ab3c-09f0c8e94387","Type":"ContainerStarted","Data":"384da626590cb504b3e9f43548d18c618bae79bf68819edbfc730c0bf3ff3d1f"} Sep 11 18:18:43.942211 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:43.942164 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fbd7656bf-jrw2n" podStartSLOduration=1.94214832 podStartE2EDuration="1.94214832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:43.940886829 +0000 UTC m=+192.167563195" watchObservedRunningTime="2026-09-11 18:18:43.94214832 +0000 UTC m=+192.168824661" Sep 11 18:18:44.811053 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:44.811019 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:18:45.627407 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:45.627374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:45.726856 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:45.726825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:45.934457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:45.934371 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerStarted","Data":"1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2"} Sep 11 18:18:45.967765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:45.967716 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.4959736599999998 podStartE2EDuration="8.967686744s" totalImagesPullingTime="5.452793435s" totalInitContainerRuntime="18.919649ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:45.966130384 +0000 UTC m=+194.192806736" watchObservedRunningTime="2026-09-11 18:18:45.967686744 +0000 UTC m=+194.194363084" Sep 11 18:18:47.210261 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:47.210208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:47.314484 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:47.314452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:47.395873 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:47.395826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:47.923445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:47.923409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:48.125986 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:48.125958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:48.227575 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:48.227495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:51.529367 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:51.529337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:51.630202 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:51.630170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:52.624073 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:52.624039 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:52.624465 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:52.624095 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:52.628873 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:52.628852 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:52.966498 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:52.966420 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:18:53.021537 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:53.021508 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f6bf575fb-ngfzn"] Sep 11 18:18:53.144905 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:53.144861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:53.247790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:53.247672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:53.349412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:53.349378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:53.968673 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:53.968641 2579 generic.go:356] "Generic (PLEG): container finished" podID="8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84" containerID="8f2dc3cd42b3b4ec404d43fce77526c6387e0a916dfa97e4cea98a3cfab7c00d" exitCode=0 Sep 11 18:18:53.969055 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:53.968729 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" event={"ID":"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84","Type":"ContainerDied","Data":"8f2dc3cd42b3b4ec404d43fce77526c6387e0a916dfa97e4cea98a3cfab7c00d"} Sep 11 18:18:53.969151 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:53.969137 2579 scope.go:117] "RemoveContainer" containerID="8f2dc3cd42b3b4ec404d43fce77526c6387e0a916dfa97e4cea98a3cfab7c00d" Sep 11 18:18:54.703181 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:54.703149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:54.805518 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:54.805483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:54.975012 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:54.974934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2xhfd" event={"ID":"8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84","Type":"ContainerStarted","Data":"b901bb7024cce5c2b1f5678f085ad07d2c39a95b6c0712bccbcc94915895e6e8"} Sep 11 18:18:58.104477 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.104413 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" podUID="050a7f57-b326-4310-8617-5075add177fb" containerName="registry" containerID="cri-o://192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e" gracePeriod=30 Sep 11 18:18:58.342838 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.342811 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:18:58.466372 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466279 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-trusted-ca\") pod \"050a7f57-b326-4310-8617-5075add177fb\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " Sep 11 18:18:58.466372 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466332 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") pod \"050a7f57-b326-4310-8617-5075add177fb\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " Sep 11 18:18:58.466372 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466374 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-installation-pull-secrets\") pod \"050a7f57-b326-4310-8617-5075add177fb\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " Sep 11 18:18:58.466619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466395 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-registry-certificates\") pod \"050a7f57-b326-4310-8617-5075add177fb\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " Sep 11 18:18:58.466619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466440 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-image-registry-private-configuration\") pod \"050a7f57-b326-4310-8617-5075add177fb\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " Sep 11 18:18:58.466619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466570 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vvp5\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-kube-api-access-7vvp5\") pod \"050a7f57-b326-4310-8617-5075add177fb\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " Sep 11 18:18:58.466817 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466629 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-bound-sa-token\") pod \"050a7f57-b326-4310-8617-5075add177fb\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " Sep 11 18:18:58.466817 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466671 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/050a7f57-b326-4310-8617-5075add177fb-ca-trust-extracted\") pod \"050a7f57-b326-4310-8617-5075add177fb\" (UID: \"050a7f57-b326-4310-8617-5075add177fb\") " Sep 11 18:18:58.466927 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466817 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "050a7f57-b326-4310-8617-5075add177fb" (UID: "050a7f57-b326-4310-8617-5075add177fb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:18:58.466927 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.466906 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "050a7f57-b326-4310-8617-5075add177fb" (UID: "050a7f57-b326-4310-8617-5075add177fb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:18:58.467039 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.467015 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-trusted-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:18:58.467039 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.467032 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/050a7f57-b326-4310-8617-5075add177fb-registry-certificates\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:18:58.468834 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.468803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "050a7f57-b326-4310-8617-5075add177fb" (UID: "050a7f57-b326-4310-8617-5075add177fb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:18:58.469015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.468986 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "050a7f57-b326-4310-8617-5075add177fb" (UID: "050a7f57-b326-4310-8617-5075add177fb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:18:58.469134 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.469107 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "050a7f57-b326-4310-8617-5075add177fb" (UID: "050a7f57-b326-4310-8617-5075add177fb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:18:58.469319 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.469296 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "050a7f57-b326-4310-8617-5075add177fb" (UID: "050a7f57-b326-4310-8617-5075add177fb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:18:58.469419 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.469383 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-kube-api-access-7vvp5" (OuterVolumeSpecName: "kube-api-access-7vvp5") pod "050a7f57-b326-4310-8617-5075add177fb" (UID: "050a7f57-b326-4310-8617-5075add177fb"). InnerVolumeSpecName "kube-api-access-7vvp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:18:58.475303 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.475280 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050a7f57-b326-4310-8617-5075add177fb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "050a7f57-b326-4310-8617-5075add177fb" (UID: "050a7f57-b326-4310-8617-5075add177fb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:18:58.568003 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.567967 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-image-registry-private-configuration\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:18:58.568003 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.567998 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvp5\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-kube-api-access-7vvp5\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:18:58.568003 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.568009 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-bound-sa-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:18:58.568212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.568020 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/050a7f57-b326-4310-8617-5075add177fb-ca-trust-extracted\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:18:58.568212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.568030 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050a7f57-b326-4310-8617-5075add177fb-registry-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:18:58.568212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.568038 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/050a7f57-b326-4310-8617-5075add177fb-installation-pull-secrets\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:18:58.989766 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.989733 2579 generic.go:356] "Generic (PLEG): container finished" podID="050a7f57-b326-4310-8617-5075add177fb" containerID="192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e" exitCode=0 Sep 11 18:18:58.989982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.989811 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" Sep 11 18:18:58.989982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.989852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" event={"ID":"050a7f57-b326-4310-8617-5075add177fb","Type":"ContainerDied","Data":"192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e"} Sep 11 18:18:58.989982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.989904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648d6bd57b-sljfs" event={"ID":"050a7f57-b326-4310-8617-5075add177fb","Type":"ContainerDied","Data":"f161267727dbe4e3d876fa08c8e2ea50eb2626ba6017a2a86eab2f109f6cc941"} Sep 11 18:18:58.990514 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:58.990496 2579 scope.go:117] "RemoveContainer" containerID="192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e" Sep 11 18:18:59.000364 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:59.000346 2579 scope.go:117] "RemoveContainer" containerID="192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e" Sep 11 18:18:59.000638 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:59.000612 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e\": container with ID starting with 192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e not found: ID does not exist" containerID="192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e" Sep 11 18:18:59.000716 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:59.000650 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e"} err="failed to get container status \"192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e\": rpc error: code = NotFound desc = could not find container \"192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e\": container with ID starting with 192a68eabdfc5bc140fed5456fc2f55a5a7b2352eeb76c05903f83fb1bcbc95e not found: ID does not exist" Sep 11 18:18:59.012905 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:18:59.012884 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050a7f57_b326_4310_8617_5075add177fb.slice\": RecentStats: unable to find data in memory cache]" Sep 11 18:18:59.013228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:59.013208 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648d6bd57b-sljfs"] Sep 11 18:18:59.016777 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:18:59.016755 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-648d6bd57b-sljfs"] Sep 11 18:19:00.227082 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:00.227047 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050a7f57-b326-4310-8617-5075add177fb" path="/var/lib/kubelet/pods/050a7f57-b326-4310-8617-5075add177fb/volumes" Sep 11 18:19:02.905565 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:02.905520 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5998f5dcd7-6nt5h" podUID="75f0cb41-f587-46e2-9b3e-609ea22c5d09" containerName="console" containerID="cri-o://25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36" gracePeriod=15 Sep 11 18:19:03.184735 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.184638 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:19:03.309865 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.309828 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmnzb\" (UniqueName: \"kubernetes.io/projected/75f0cb41-f587-46e2-9b3e-609ea22c5d09-kube-api-access-lmnzb\") pod \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " Sep 11 18:19:03.310020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.309883 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-service-ca\") pod \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " Sep 11 18:19:03.310020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.309933 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-serving-cert\") pod \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " Sep 11 18:19:03.310020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.309961 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-oauth-serving-cert\") pod \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " Sep 11 18:19:03.310020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.310000 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-config\") pod \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " Sep 11 18:19:03.310169 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.310035 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-oauth-config\") pod \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\" (UID: \"75f0cb41-f587-46e2-9b3e-609ea22c5d09\") " Sep 11 18:19:03.310405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.310369 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-service-ca" (OuterVolumeSpecName: "service-ca") pod "75f0cb41-f587-46e2-9b3e-609ea22c5d09" (UID: "75f0cb41-f587-46e2-9b3e-609ea22c5d09"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:03.310405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.310386 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-config" (OuterVolumeSpecName: "console-config") pod "75f0cb41-f587-46e2-9b3e-609ea22c5d09" (UID: "75f0cb41-f587-46e2-9b3e-609ea22c5d09"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:03.310562 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.310370 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "75f0cb41-f587-46e2-9b3e-609ea22c5d09" (UID: "75f0cb41-f587-46e2-9b3e-609ea22c5d09"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:03.312128 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.312106 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "75f0cb41-f587-46e2-9b3e-609ea22c5d09" (UID: "75f0cb41-f587-46e2-9b3e-609ea22c5d09"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:03.312206 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.312183 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "75f0cb41-f587-46e2-9b3e-609ea22c5d09" (UID: "75f0cb41-f587-46e2-9b3e-609ea22c5d09"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:03.312242 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.312221 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f0cb41-f587-46e2-9b3e-609ea22c5d09-kube-api-access-lmnzb" (OuterVolumeSpecName: "kube-api-access-lmnzb") pod "75f0cb41-f587-46e2-9b3e-609ea22c5d09" (UID: "75f0cb41-f587-46e2-9b3e-609ea22c5d09"). InnerVolumeSpecName "kube-api-access-lmnzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:19:03.411210 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.411173 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:03.411210 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.411203 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:03.411210 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.411212 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-oauth-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:03.411210 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.411221 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:03.411469 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.411230 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f0cb41-f587-46e2-9b3e-609ea22c5d09-console-oauth-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:03.411469 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:03.411238 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmnzb\" (UniqueName: \"kubernetes.io/projected/75f0cb41-f587-46e2-9b3e-609ea22c5d09-kube-api-access-lmnzb\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:04.010144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.010106 2579 generic.go:356] "Generic (PLEG): container finished" podID="75f0cb41-f587-46e2-9b3e-609ea22c5d09" containerID="25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36" exitCode=0 Sep 11 18:19:04.010559 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.010153 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5998f5dcd7-6nt5h" Sep 11 18:19:04.010559 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.010195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5998f5dcd7-6nt5h" event={"ID":"75f0cb41-f587-46e2-9b3e-609ea22c5d09","Type":"ContainerDied","Data":"25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36"} Sep 11 18:19:04.010559 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.010238 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5998f5dcd7-6nt5h" event={"ID":"75f0cb41-f587-46e2-9b3e-609ea22c5d09","Type":"ContainerDied","Data":"daf4f093dffa6d7d5a2eaaf44e90a606c661dfef4ee3d5727b1a59011d2a45e7"} Sep 11 18:19:04.010559 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.010256 2579 scope.go:117] "RemoveContainer" containerID="25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36" Sep 11 18:19:04.019035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.019017 2579 scope.go:117] "RemoveContainer" containerID="25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36" Sep 11 18:19:04.019322 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:19:04.019299 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36\": container with ID starting with 25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36 not found: ID does not exist" containerID="25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36" Sep 11 18:19:04.019370 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.019330 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36"} err="failed to get container status \"25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36\": rpc error: code = NotFound desc = could not find container \"25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36\": container with ID starting with 25d1c91054d883b84fd2ee23cbc3bec3c40c8c94cf3fbb7e40695f5aa5247b36 not found: ID does not exist" Sep 11 18:19:04.030326 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.030300 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5998f5dcd7-6nt5h"] Sep 11 18:19:04.034230 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.034210 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5998f5dcd7-6nt5h"] Sep 11 18:19:04.227375 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:04.227345 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f0cb41-f587-46e2-9b3e-609ea22c5d09" path="/var/lib/kubelet/pods/75f0cb41-f587-46e2-9b3e-609ea22c5d09/volumes" Sep 11 18:19:06.526069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:06.526036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:06.591004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:06.590973 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:18.046363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.046316 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f6bf575fb-ngfzn" podUID="37b49af5-7b8e-4179-86b0-43209bf6db7f" containerName="console" containerID="cri-o://d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450" gracePeriod=15 Sep 11 18:19:18.317605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.317535 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:19:18.446256 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446221 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-oauth-config\") pod \"37b49af5-7b8e-4179-86b0-43209bf6db7f\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " Sep 11 18:19:18.446256 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446262 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-service-ca\") pod \"37b49af5-7b8e-4179-86b0-43209bf6db7f\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " Sep 11 18:19:18.446475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446290 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-serving-cert\") pod \"37b49af5-7b8e-4179-86b0-43209bf6db7f\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " Sep 11 18:19:18.446475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446405 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-trusted-ca-bundle\") pod \"37b49af5-7b8e-4179-86b0-43209bf6db7f\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " Sep 11 18:19:18.446475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446446 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpwnt\" (UniqueName: \"kubernetes.io/projected/37b49af5-7b8e-4179-86b0-43209bf6db7f-kube-api-access-hpwnt\") pod \"37b49af5-7b8e-4179-86b0-43209bf6db7f\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " Sep 11 18:19:18.446475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446475 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-oauth-serving-cert\") pod \"37b49af5-7b8e-4179-86b0-43209bf6db7f\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " Sep 11 18:19:18.446724 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446542 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-config\") pod \"37b49af5-7b8e-4179-86b0-43209bf6db7f\" (UID: \"37b49af5-7b8e-4179-86b0-43209bf6db7f\") " Sep 11 18:19:18.446819 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446794 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-service-ca" (OuterVolumeSpecName: "service-ca") pod "37b49af5-7b8e-4179-86b0-43209bf6db7f" (UID: "37b49af5-7b8e-4179-86b0-43209bf6db7f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:18.446903 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446879 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "37b49af5-7b8e-4179-86b0-43209bf6db7f" (UID: "37b49af5-7b8e-4179-86b0-43209bf6db7f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:18.446953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.446924 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "37b49af5-7b8e-4179-86b0-43209bf6db7f" (UID: "37b49af5-7b8e-4179-86b0-43209bf6db7f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:18.447148 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.447126 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-config" (OuterVolumeSpecName: "console-config") pod "37b49af5-7b8e-4179-86b0-43209bf6db7f" (UID: "37b49af5-7b8e-4179-86b0-43209bf6db7f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:18.448466 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.448441 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "37b49af5-7b8e-4179-86b0-43209bf6db7f" (UID: "37b49af5-7b8e-4179-86b0-43209bf6db7f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:18.448576 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.448476 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b49af5-7b8e-4179-86b0-43209bf6db7f-kube-api-access-hpwnt" (OuterVolumeSpecName: "kube-api-access-hpwnt") pod "37b49af5-7b8e-4179-86b0-43209bf6db7f" (UID: "37b49af5-7b8e-4179-86b0-43209bf6db7f"). InnerVolumeSpecName "kube-api-access-hpwnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:19:18.448576 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.448496 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "37b49af5-7b8e-4179-86b0-43209bf6db7f" (UID: "37b49af5-7b8e-4179-86b0-43209bf6db7f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:18.547351 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.547318 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:18.547351 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.547345 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-oauth-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:18.547351 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.547355 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:18.547581 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.547365 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37b49af5-7b8e-4179-86b0-43209bf6db7f-console-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:18.547581 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.547376 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-trusted-ca-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:18.547581 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.547386 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpwnt\" (UniqueName: \"kubernetes.io/projected/37b49af5-7b8e-4179-86b0-43209bf6db7f-kube-api-access-hpwnt\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:18.547581 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:18.547394 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37b49af5-7b8e-4179-86b0-43209bf6db7f-oauth-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:19.071323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:19.071285 2579 generic.go:356] "Generic (PLEG): container finished" podID="37b49af5-7b8e-4179-86b0-43209bf6db7f" containerID="d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450" exitCode=0 Sep 11 18:19:19.071754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:19.071333 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6bf575fb-ngfzn" Sep 11 18:19:19.071754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:19.071370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6bf575fb-ngfzn" event={"ID":"37b49af5-7b8e-4179-86b0-43209bf6db7f","Type":"ContainerDied","Data":"d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450"} Sep 11 18:19:19.071754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:19.071425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6bf575fb-ngfzn" event={"ID":"37b49af5-7b8e-4179-86b0-43209bf6db7f","Type":"ContainerDied","Data":"854d214dbbf9946f0c346124d5060fd41ae877e84093a8c6279c1406dacd3c4a"} Sep 11 18:19:19.071754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:19.071446 2579 scope.go:117] "RemoveContainer" containerID="d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450" Sep 11 18:19:19.079719 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:19.079688 2579 scope.go:117] "RemoveContainer" containerID="d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450" Sep 11 18:19:19.080007 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:19:19.079984 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450\": container with ID starting with d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450 not found: ID does not exist" containerID="d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450" Sep 11 18:19:19.080099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:19.080013 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450"} err="failed to get container status \"d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450\": rpc error: code = NotFound desc = could not find container \"d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450\": container with ID starting with d57072182910bb4ab5b3fedd2b3b8929fc0ccf3f9502f3b70f04bd328cc25450 not found: ID does not exist" Sep 11 18:19:19.090526 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:19.090505 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f6bf575fb-ngfzn"] Sep 11 18:19:19.094056 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:19.094037 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f6bf575fb-ngfzn"] Sep 11 18:19:20.227169 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:20.227137 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b49af5-7b8e-4179-86b0-43209bf6db7f" path="/var/lib/kubelet/pods/37b49af5-7b8e-4179-86b0-43209bf6db7f/volumes" Sep 11 18:19:26.850280 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:26.850242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:44.151261 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:44.151219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:19:44.153446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:44.153425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03fbffc5-9471-44da-b5b0-4b881b17898c-metrics-certs\") pod \"network-metrics-daemon-bw29p\" (UID: \"03fbffc5-9471-44da-b5b0-4b881b17898c\") " pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:19:44.326333 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:44.326299 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kq7d5\"" Sep 11 18:19:44.333454 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:44.333435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw29p" Sep 11 18:19:44.602621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:44.602431 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bw29p"] Sep 11 18:19:44.605309 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:19:44.605278 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fbffc5_9471_44da_b5b0_4b881b17898c.slice/crio-864885628afc16254533538c4c7f3c766d739ff41d8f1b9d0691c879856086c0 WatchSource:0}: Error finding container 864885628afc16254533538c4c7f3c766d739ff41d8f1b9d0691c879856086c0: Status 404 returned error can't find the container with id 864885628afc16254533538c4c7f3c766d739ff41d8f1b9d0691c879856086c0 Sep 11 18:19:45.170271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:45.170232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bw29p" event={"ID":"03fbffc5-9471-44da-b5b0-4b881b17898c","Type":"ContainerStarted","Data":"864885628afc16254533538c4c7f3c766d739ff41d8f1b9d0691c879856086c0"} Sep 11 18:19:46.175111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:46.175072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bw29p" event={"ID":"03fbffc5-9471-44da-b5b0-4b881b17898c","Type":"ContainerStarted","Data":"3093f60b8edf83eaad7b365424ef5a41a497f7ddb1304dc64c3d92a1423d0c07"} Sep 11 18:19:46.175551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:46.175168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bw29p" event={"ID":"03fbffc5-9471-44da-b5b0-4b881b17898c","Type":"ContainerStarted","Data":"91c8b16304d3b4e6be6377a211ea6300aed47429dbc190746918380059acd830"} Sep 11 18:19:46.194578 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:46.194527 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bw29p" podStartSLOduration=253.406584018 podStartE2EDuration="4m14.194512573s" totalImagesPullingTime="787.928555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:46.193035092 +0000 UTC m=+254.419711443" watchObservedRunningTime="2026-09-11 18:19:46.194512573 +0000 UTC m=+254.421188910" Sep 11 18:19:46.290423 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:46.290391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:46.358620 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:46.358580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:50.015513 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:50.015474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:50.073736 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:50.073680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:51.583126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:51.583093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:51.647323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:51.647289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:55.745635 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:55.745541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:55.860592 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:55.860550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:56.786832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:56.786797 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 18:19:56.787237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:56.787063 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="alertmanager" containerID="cri-o://a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57" gracePeriod=120 Sep 11 18:19:56.787237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:56.787125 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy-metric" containerID="cri-o://a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f" gracePeriod=120 Sep 11 18:19:56.787237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:56.787146 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="prom-label-proxy" containerID="cri-o://1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2" gracePeriod=120 Sep 11 18:19:56.787237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:56.787172 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy-web" containerID="cri-o://ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a" gracePeriod=120 Sep 11 18:19:56.787237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:56.787202 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="config-reloader" containerID="cri-o://30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b" gracePeriod=120 Sep 11 18:19:56.787446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:56.787238 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy" containerID="cri-o://f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa" gracePeriod=120 Sep 11 18:19:57.220934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:57.220844 2579 generic.go:356] "Generic (PLEG): container finished" podID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerID="1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2" exitCode=0 Sep 11 18:19:57.220934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:57.220871 2579 generic.go:356] "Generic (PLEG): container finished" podID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerID="f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa" exitCode=0 Sep 11 18:19:57.220934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:57.220878 2579 generic.go:356] "Generic (PLEG): container finished" podID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerID="30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b" exitCode=0 Sep 11 18:19:57.220934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:57.220884 2579 generic.go:356] "Generic (PLEG): container finished" podID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerID="a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57" exitCode=0 Sep 11 18:19:57.220934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:57.220923 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerDied","Data":"1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2"} Sep 11 18:19:57.221234 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:57.221002 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerDied","Data":"f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa"} Sep 11 18:19:57.221234 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:57.221041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerDied","Data":"30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b"} Sep 11 18:19:57.221234 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:57.221076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerDied","Data":"a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57"} Sep 11 18:19:58.029624 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.029600 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.052882 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.052859 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-main-tls\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.052894 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-cluster-tls-config\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.052913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-trusted-ca-bundle\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.052943 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.052968 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-metrics-client-ca\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053267 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053102 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-web-config\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053267 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-main-db\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053267 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053179 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-tls-assets\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053267 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053211 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-out\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053461 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053278 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053461 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053313 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:58.053461 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-web\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053461 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053322 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:58.053461 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trfnv\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-kube-api-access-trfnv\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053461 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053384 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-volume\") pod \"6383cc78-8370-4ab6-a91c-938bb4ec690f\" (UID: \"6383cc78-8370-4ab6-a91c-938bb4ec690f\") " Sep 11 18:19:58.053767 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053594 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.053767 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.053611 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6383cc78-8370-4ab6-a91c-938bb4ec690f-metrics-client-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.054388 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.054004 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:19:58.056671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.056435 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:58.056671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.056596 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:58.056671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.056635 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:58.057462 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.057433 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:19:58.057581 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.057515 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-volume" (OuterVolumeSpecName: "config-volume") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:58.058438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.058062 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-kube-api-access-trfnv" (OuterVolumeSpecName: "kube-api-access-trfnv") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "kube-api-access-trfnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:19:58.058438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.058381 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:58.058933 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.058900 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-out" (OuterVolumeSpecName: "config-out") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:19:58.061047 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.061019 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:58.068916 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.068856 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-web-config" (OuterVolumeSpecName: "web-config") pod "6383cc78-8370-4ab6-a91c-938bb4ec690f" (UID: "6383cc78-8370-4ab6-a91c-938bb4ec690f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:58.154318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154231 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154260 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trfnv\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-kube-api-access-trfnv\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154273 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-volume\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154282 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-main-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154292 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-cluster-tls-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154300 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154309 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-web-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154663 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154317 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-alertmanager-main-db\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154663 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154339 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6383cc78-8370-4ab6-a91c-938bb4ec690f-tls-assets\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154663 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154348 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6383cc78-8370-4ab6-a91c-938bb4ec690f-config-out\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.154663 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.154356 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6383cc78-8370-4ab6-a91c-938bb4ec690f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:19:58.228447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.228414 2579 generic.go:356] "Generic (PLEG): container finished" podID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerID="a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f" exitCode=0 Sep 11 18:19:58.228447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.228440 2579 generic.go:356] "Generic (PLEG): container finished" podID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerID="ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a" exitCode=0 Sep 11 18:19:58.228675 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.228494 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.228675 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.228493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerDied","Data":"a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f"} Sep 11 18:19:58.228675 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.228630 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerDied","Data":"ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a"} Sep 11 18:19:58.228675 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.228661 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6383cc78-8370-4ab6-a91c-938bb4ec690f","Type":"ContainerDied","Data":"98ff1f3de8bfee2884276940dc0003b356c931ee84ecb694731372aabb0c77b4"} Sep 11 18:19:58.228884 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.228698 2579 scope.go:117] "RemoveContainer" containerID="1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2" Sep 11 18:19:58.236631 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.236600 2579 scope.go:117] "RemoveContainer" containerID="a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f" Sep 11 18:19:58.244173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.244157 2579 scope.go:117] "RemoveContainer" containerID="f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa" Sep 11 18:19:58.250551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.250535 2579 scope.go:117] "RemoveContainer" containerID="ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a" Sep 11 18:19:58.255292 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.255266 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 18:19:58.257366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.257350 2579 scope.go:117] "RemoveContainer" containerID="30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b" Sep 11 18:19:58.260194 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.260172 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 18:19:58.264824 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.264806 2579 scope.go:117] "RemoveContainer" containerID="a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57" Sep 11 18:19:58.271335 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.271320 2579 scope.go:117] "RemoveContainer" containerID="d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4" Sep 11 18:19:58.277731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.277715 2579 scope.go:117] "RemoveContainer" containerID="1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2" Sep 11 18:19:58.277985 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:19:58.277969 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2\": container with ID starting with 1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2 not found: ID does not exist" containerID="1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2" Sep 11 18:19:58.278035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.277991 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2"} err="failed to get container status \"1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2\": rpc error: code = NotFound desc = could not find container \"1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2\": container with ID starting with 1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2 not found: ID does not exist" Sep 11 18:19:58.278035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.278009 2579 scope.go:117] "RemoveContainer" containerID="a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f" Sep 11 18:19:58.278230 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:19:58.278216 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f\": container with ID starting with a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f not found: ID does not exist" containerID="a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f" Sep 11 18:19:58.278271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.278234 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f"} err="failed to get container status \"a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f\": rpc error: code = NotFound desc = could not find container \"a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f\": container with ID starting with a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f not found: ID does not exist" Sep 11 18:19:58.278271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.278248 2579 scope.go:117] "RemoveContainer" containerID="f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa" Sep 11 18:19:58.278496 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:19:58.278478 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa\": container with ID starting with f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa not found: ID does not exist" containerID="f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa" Sep 11 18:19:58.278533 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.278504 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa"} err="failed to get container status \"f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa\": rpc error: code = NotFound desc = could not find container \"f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa\": container with ID starting with f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa not found: ID does not exist" Sep 11 18:19:58.278533 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.278521 2579 scope.go:117] "RemoveContainer" containerID="ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a" Sep 11 18:19:58.278771 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:19:58.278748 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a\": container with ID starting with ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a not found: ID does not exist" containerID="ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a" Sep 11 18:19:58.278809 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.278779 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a"} err="failed to get container status \"ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a\": rpc error: code = NotFound desc = could not find container \"ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a\": container with ID starting with ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a not found: ID does not exist" Sep 11 18:19:58.278809 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.278802 2579 scope.go:117] "RemoveContainer" containerID="30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b" Sep 11 18:19:58.279066 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:19:58.279050 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b\": container with ID starting with 30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b not found: ID does not exist" containerID="30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b" Sep 11 18:19:58.279110 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.279070 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b"} err="failed to get container status \"30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b\": rpc error: code = NotFound desc = could not find container \"30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b\": container with ID starting with 30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b not found: ID does not exist" Sep 11 18:19:58.279110 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.279083 2579 scope.go:117] "RemoveContainer" containerID="a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57" Sep 11 18:19:58.279327 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:19:58.279310 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57\": container with ID starting with a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57 not found: ID does not exist" containerID="a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57" Sep 11 18:19:58.279776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.279450 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57"} err="failed to get container status \"a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57\": rpc error: code = NotFound desc = could not find container \"a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57\": container with ID starting with a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57 not found: ID does not exist" Sep 11 18:19:58.279776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.279677 2579 scope.go:117] "RemoveContainer" containerID="d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4" Sep 11 18:19:58.284189 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:19:58.284165 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4\": container with ID starting with d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4 not found: ID does not exist" containerID="d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4" Sep 11 18:19:58.284285 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.284195 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4"} err="failed to get container status \"d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4\": rpc error: code = NotFound desc = could not find container \"d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4\": container with ID starting with d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4 not found: ID does not exist" Sep 11 18:19:58.284285 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.284215 2579 scope.go:117] "RemoveContainer" containerID="1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2" Sep 11 18:19:58.284491 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.284472 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2"} err="failed to get container status \"1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2\": rpc error: code = NotFound desc = could not find container \"1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2\": container with ID starting with 1f22080ccce0b4553b66f7cd8a4237ff96423bcb39bb94b2c6594844a54b12d2 not found: ID does not exist" Sep 11 18:19:58.284546 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.284493 2579 scope.go:117] "RemoveContainer" containerID="a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f" Sep 11 18:19:58.284835 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.284816 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f"} err="failed to get container status \"a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f\": rpc error: code = NotFound desc = could not find container \"a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f\": container with ID starting with a771975c29a012184ead9221518506533315913a6382aa03744c1badd3a8cc6f not found: ID does not exist" Sep 11 18:19:58.284835 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.284835 2579 scope.go:117] "RemoveContainer" containerID="f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa" Sep 11 18:19:58.285104 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.285079 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa"} err="failed to get container status \"f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa\": rpc error: code = NotFound desc = could not find container \"f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa\": container with ID starting with f6da0da0af2892a48ed7be5716e7849c0cf1cec743f9a0b0c1bbfec5fe0e39aa not found: ID does not exist" Sep 11 18:19:58.285104 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.285098 2579 scope.go:117] "RemoveContainer" containerID="ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a" Sep 11 18:19:58.285346 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.285329 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a"} err="failed to get container status \"ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a\": rpc error: code = NotFound desc = could not find container \"ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a\": container with ID starting with ecb896de7d7f671cb4b4b4d9c646285cc4d9a30979ce8fbcb35dcca20c1dfb5a not found: ID does not exist" Sep 11 18:19:58.285398 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.285347 2579 scope.go:117] "RemoveContainer" containerID="30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b" Sep 11 18:19:58.285591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.285559 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b"} err="failed to get container status \"30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b\": rpc error: code = NotFound desc = could not find container \"30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b\": container with ID starting with 30a9b3db4d9e8b52de6e8991ed437546d9db2ced12ba68b01b3b25abc6d4a51b not found: ID does not exist" Sep 11 18:19:58.285642 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.285595 2579 scope.go:117] "RemoveContainer" containerID="a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57" Sep 11 18:19:58.285926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.285897 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57"} err="failed to get container status \"a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57\": rpc error: code = NotFound desc = could not find container \"a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57\": container with ID starting with a5fad962a57f3a11a157f8b8863131d34c9d2dbe1c617bef73f9ce58fb438d57 not found: ID does not exist" Sep 11 18:19:58.285990 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.285928 2579 scope.go:117] "RemoveContainer" containerID="d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4" Sep 11 18:19:58.286246 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.286227 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4"} err="failed to get container status \"d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4\": rpc error: code = NotFound desc = could not find container \"d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4\": container with ID starting with d7fc0010e89dd8bd380404343c628c5aa63a2e44187a4282d30b86f1c3b584a4 not found: ID does not exist" Sep 11 18:19:58.289420 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289400 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 18:19:58.289662 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289649 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b49af5-7b8e-4179-86b0-43209bf6db7f" containerName="console" Sep 11 18:19:58.289725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289664 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b49af5-7b8e-4179-86b0-43209bf6db7f" containerName="console" Sep 11 18:19:58.289725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289680 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy-metric" Sep 11 18:19:58.289725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289685 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy-metric" Sep 11 18:19:58.289725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289692 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy" Sep 11 18:19:58.289725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289698 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy" Sep 11 18:19:58.289725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289724 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy-web" Sep 11 18:19:58.289725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289729 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy-web" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289735 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="prom-label-proxy" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289740 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="prom-label-proxy" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289747 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f0cb41-f587-46e2-9b3e-609ea22c5d09" containerName="console" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289753 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f0cb41-f587-46e2-9b3e-609ea22c5d09" containerName="console" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289760 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050a7f57-b326-4310-8617-5075add177fb" containerName="registry" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289766 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="050a7f57-b326-4310-8617-5075add177fb" containerName="registry" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289772 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="init-config-reloader" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289776 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="init-config-reloader" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289782 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="alertmanager" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289787 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="alertmanager" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289792 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="config-reloader" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289797 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="config-reloader" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289836 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f0cb41-f587-46e2-9b3e-609ea22c5d09" containerName="console" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289844 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="alertmanager" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289850 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b49af5-7b8e-4179-86b0-43209bf6db7f" containerName="console" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289855 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="prom-label-proxy" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289861 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="config-reloader" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289866 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289871 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy-metric" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289878 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="050a7f57-b326-4310-8617-5075add177fb" containerName="registry" Sep 11 18:19:58.289967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.289884 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" containerName="kube-rbac-proxy-web" Sep 11 18:19:58.293409 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.293395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.295734 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.295681 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 18:19:58.295830 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.295737 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 18:19:58.295896 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.295878 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 18:19:58.295958 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.295878 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 18:19:58.296024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.296005 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 18:19:58.296172 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.296149 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 18:19:58.296263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.296204 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 18:19:58.296263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.296215 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 18:19:58.296263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.296242 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dslx4\"" Sep 11 18:19:58.301569 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.301546 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 18:19:58.309672 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.309651 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 18:19:58.355553 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4799c69c-32cc-4d73-8c4f-28627aa7981a-config-out\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smvrk\" (UniqueName: \"kubernetes.io/projected/4799c69c-32cc-4d73-8c4f-28627aa7981a-kube-api-access-smvrk\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-web-config\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4799c69c-32cc-4d73-8c4f-28627aa7981a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4799c69c-32cc-4d73-8c4f-28627aa7981a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355847 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355847 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355847 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4799c69c-32cc-4d73-8c4f-28627aa7981a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355847 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355847 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.355847 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-config-volume\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.356021 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.356021 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.355903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4799c69c-32cc-4d73-8c4f-28627aa7981a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.456926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.456853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4799c69c-32cc-4d73-8c4f-28627aa7981a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.456926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.456886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.456926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.456903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.456926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.456922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-config-volume\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4799c69c-32cc-4d73-8c4f-28627aa7981a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4799c69c-32cc-4d73-8c4f-28627aa7981a-config-out\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smvrk\" (UniqueName: \"kubernetes.io/projected/4799c69c-32cc-4d73-8c4f-28627aa7981a-kube-api-access-smvrk\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-web-config\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4799c69c-32cc-4d73-8c4f-28627aa7981a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4799c69c-32cc-4d73-8c4f-28627aa7981a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.457889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.457861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4799c69c-32cc-4d73-8c4f-28627aa7981a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.458167 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.458141 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4799c69c-32cc-4d73-8c4f-28627aa7981a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.459849 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.459806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4799c69c-32cc-4d73-8c4f-28627aa7981a-config-out\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.459951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.459933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4799c69c-32cc-4d73-8c4f-28627aa7981a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.460132 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.460107 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.460213 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.460195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.460353 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.460338 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4799c69c-32cc-4d73-8c4f-28627aa7981a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.460395 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.460343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.460718 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.460676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.460799 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.460767 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-config-volume\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.460977 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.460958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-web-config\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.461509 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.461492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4799c69c-32cc-4d73-8c4f-28627aa7981a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.466772 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.466752 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smvrk\" (UniqueName: \"kubernetes.io/projected/4799c69c-32cc-4d73-8c4f-28627aa7981a-kube-api-access-smvrk\") pod \"alertmanager-main-0\" (UID: \"4799c69c-32cc-4d73-8c4f-28627aa7981a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.602304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.602272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 18:19:58.880135 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:58.880111 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 18:19:58.883663 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:19:58.883629 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4799c69c_32cc_4d73_8c4f_28627aa7981a.slice/crio-b103e0a6b253329adcf37a57fb7518743880bbf56624b034dbcc4b44012360e2 WatchSource:0}: Error finding container b103e0a6b253329adcf37a57fb7518743880bbf56624b034dbcc4b44012360e2: Status 404 returned error can't find the container with id b103e0a6b253329adcf37a57fb7518743880bbf56624b034dbcc4b44012360e2 Sep 11 18:19:59.234581 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:59.234496 2579 generic.go:356] "Generic (PLEG): container finished" podID="4799c69c-32cc-4d73-8c4f-28627aa7981a" containerID="40fa44b9dbde47075596a9004ae88d2afa6b3ca5cad655b5b809834cbb7356dd" exitCode=0 Sep 11 18:19:59.234581 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:59.234564 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4799c69c-32cc-4d73-8c4f-28627aa7981a","Type":"ContainerDied","Data":"40fa44b9dbde47075596a9004ae88d2afa6b3ca5cad655b5b809834cbb7356dd"} Sep 11 18:19:59.234981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:19:59.234628 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4799c69c-32cc-4d73-8c4f-28627aa7981a","Type":"ContainerStarted","Data":"b103e0a6b253329adcf37a57fb7518743880bbf56624b034dbcc4b44012360e2"} Sep 11 18:20:00.228665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.228569 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6383cc78-8370-4ab6-a91c-938bb4ec690f" path="/var/lib/kubelet/pods/6383cc78-8370-4ab6-a91c-938bb4ec690f/volumes" Sep 11 18:20:00.241294 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.241259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4799c69c-32cc-4d73-8c4f-28627aa7981a","Type":"ContainerStarted","Data":"d3d24f6d72d56cf86319ac501733cd4deffad06da60ebd9eec7e4ca4fcbe8e37"} Sep 11 18:20:00.241730 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.241319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4799c69c-32cc-4d73-8c4f-28627aa7981a","Type":"ContainerStarted","Data":"4eba970778c7207650f617b89b5a2da51cda4d26e7850432158aa0eb10298ea1"} Sep 11 18:20:00.241730 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.241345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4799c69c-32cc-4d73-8c4f-28627aa7981a","Type":"ContainerStarted","Data":"f89c6e3bf8e51c1b59b0d207c700516e3eccbccb5aad5b26b6ebf6f32e97d704"} Sep 11 18:20:00.241730 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.241369 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4799c69c-32cc-4d73-8c4f-28627aa7981a","Type":"ContainerStarted","Data":"0d4b8f63a64697e7843f96826db09bf1f36420984fdb6dda5d7ef73c262b3a2a"} Sep 11 18:20:00.241730 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.241391 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4799c69c-32cc-4d73-8c4f-28627aa7981a","Type":"ContainerStarted","Data":"e0eb3d35823b8cf52a1b549e7783aef2c00a0b448d850a5c12a6e1f17f0e6578"} Sep 11 18:20:00.241730 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.241413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4799c69c-32cc-4d73-8c4f-28627aa7981a","Type":"ContainerStarted","Data":"dc7306223e09bee9265298d17eb1e16f6abab09712c08341f627b1943d6fb954"} Sep 11 18:20:00.268980 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.268926 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.25751329 podStartE2EDuration="2.268907019s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.393729ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:00.267082954 +0000 UTC m=+268.493759306" watchObservedRunningTime="2026-09-11 18:20:00.268907019 +0000 UTC m=+268.495583359" Sep 11 18:20:00.837072 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.837037 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-794746b5b4-4n628"] Sep 11 18:20:00.839921 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.839898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.842589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.842459 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 18:20:00.842589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.842488 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 18:20:00.842589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.842503 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 18:20:00.842589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.842512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 18:20:00.842589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.842531 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 18:20:00.842589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.842531 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nj8xj\"" Sep 11 18:20:00.847527 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.847508 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 18:20:00.853142 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.853119 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-794746b5b4-4n628"] Sep 11 18:20:00.876802 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.876779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-src9b\" (UniqueName: \"kubernetes.io/projected/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-kube-api-access-src9b\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.876944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.876847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-serving-certs-ca-bundle\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.876944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.876874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-federate-client-tls\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.876944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.876906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-secret-telemeter-client\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.876944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.876929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-telemeter-client-tls\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.877112 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.876950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-metrics-client-ca\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.877112 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.876987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.877112 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.877055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.977628 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.977590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-serving-certs-ca-bundle\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.977628 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.977630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-federate-client-tls\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.977853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.977688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-secret-telemeter-client\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.977853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.977745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-telemeter-client-tls\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.977853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.977764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-metrics-client-ca\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.977853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.977800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.978069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.977941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.978069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.978028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-src9b\" (UniqueName: \"kubernetes.io/projected/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-kube-api-access-src9b\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.978511 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.978490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-serving-certs-ca-bundle\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.978592 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.978574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-metrics-client-ca\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.978679 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.978660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.980170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.980148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-secret-telemeter-client\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.980348 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.980329 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-federate-client-tls\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.980641 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.980621 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.980769 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.980755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-telemeter-client-tls\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:00.987469 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:00.987449 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-src9b\" (UniqueName: \"kubernetes.io/projected/7081f1b8-b49f-43f0-a2f2-4abdc52f75cf-kube-api-access-src9b\") pod \"telemeter-client-794746b5b4-4n628\" (UID: \"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf\") " pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:01.150363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:01.150270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" Sep 11 18:20:01.430209 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:01.430122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-794746b5b4-4n628"] Sep 11 18:20:01.433689 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:20:01.433665 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7081f1b8_b49f_43f0_a2f2_4abdc52f75cf.slice/crio-19a468998e186dba68d431f42d3f413dae268541ecbda99cd631824dc2c005a3 WatchSource:0}: Error finding container 19a468998e186dba68d431f42d3f413dae268541ecbda99cd631824dc2c005a3: Status 404 returned error can't find the container with id 19a468998e186dba68d431f42d3f413dae268541ecbda99cd631824dc2c005a3 Sep 11 18:20:02.250438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:02.250404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" event={"ID":"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf","Type":"ContainerStarted","Data":"19a468998e186dba68d431f42d3f413dae268541ecbda99cd631824dc2c005a3"} Sep 11 18:20:03.256075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:03.256046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" event={"ID":"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf","Type":"ContainerStarted","Data":"72c0a458a25ac627120b79a61421c8ec0d83beceb1465dc869defe290fbe503d"} Sep 11 18:20:03.256360 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:03.256102 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" event={"ID":"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf","Type":"ContainerStarted","Data":"ea38a79aaeb91c33437ba4574d40dc6ca14f858dae0ef102844dbdf6198da63f"} Sep 11 18:20:04.263170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.263133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" event={"ID":"7081f1b8-b49f-43f0-a2f2-4abdc52f75cf","Type":"ContainerStarted","Data":"530ad21c58053a74787791774d713224128f0990068d91a36821aed8f79ef3ea"} Sep 11 18:20:04.288489 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.288436 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-794746b5b4-4n628" podStartSLOduration=2.597126195 podStartE2EDuration="4.288418382s" totalImagesPullingTime="1.691292187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:04.286143042 +0000 UTC m=+272.512819382" watchObservedRunningTime="2026-09-11 18:20:04.288418382 +0000 UTC m=+272.515094722" Sep 11 18:20:04.376723 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.376676 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4fff9697-spwr2"] Sep 11 18:20:04.378773 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.378747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.391083 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.391051 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4fff9697-spwr2"] Sep 11 18:20:04.408718 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.408674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9q89\" (UniqueName: \"kubernetes.io/projected/af68fbed-4f8e-40dc-8f68-d5570770e214-kube-api-access-v9q89\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.408866 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.408765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-console-config\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.408866 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.408830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-service-ca\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.408942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.408891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-serving-cert\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.408942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.408908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-oauth-serving-cert\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.408942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.408926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-oauth-config\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.408942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.408941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-trusted-ca-bundle\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.451425 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.451390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:04.509538 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.509507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-serving-cert\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.509538 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.509543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-oauth-serving-cert\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.509782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.509564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-oauth-config\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.509782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.509581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-trusted-ca-bundle\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.509782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.509612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9q89\" (UniqueName: \"kubernetes.io/projected/af68fbed-4f8e-40dc-8f68-d5570770e214-kube-api-access-v9q89\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.509782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.509641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-console-config\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.509782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.509657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-service-ca\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.510350 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.510325 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-oauth-serving-cert\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.510457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.510404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-console-config\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.510501 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.510473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-trusted-ca-bundle\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.510501 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.510495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-service-ca\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.512649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.512625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-oauth-config\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.512886 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.512863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-serving-cert\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.518542 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.518480 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9q89\" (UniqueName: \"kubernetes.io/projected/af68fbed-4f8e-40dc-8f68-d5570770e214-kube-api-access-v9q89\") pod \"console-6c4fff9697-spwr2\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.688292 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.688255 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:04.967048 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:04.966962 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4fff9697-spwr2"] Sep 11 18:20:04.970186 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:20:04.970154 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf68fbed_4f8e_40dc_8f68_d5570770e214.slice/crio-f00e8779f46969afa92f7033999018a35518fd802cf651481e2ac220f41ae4d2 WatchSource:0}: Error finding container f00e8779f46969afa92f7033999018a35518fd802cf651481e2ac220f41ae4d2: Status 404 returned error can't find the container with id f00e8779f46969afa92f7033999018a35518fd802cf651481e2ac220f41ae4d2 Sep 11 18:20:05.269330 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:05.269299 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4fff9697-spwr2" event={"ID":"af68fbed-4f8e-40dc-8f68-d5570770e214","Type":"ContainerStarted","Data":"e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04"} Sep 11 18:20:05.269685 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:05.269344 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4fff9697-spwr2" event={"ID":"af68fbed-4f8e-40dc-8f68-d5570770e214","Type":"ContainerStarted","Data":"f00e8779f46969afa92f7033999018a35518fd802cf651481e2ac220f41ae4d2"} Sep 11 18:20:05.287205 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:05.287156 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c4fff9697-spwr2" podStartSLOduration=1.287140285 podStartE2EDuration="1.287140285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:05.286557577 +0000 UTC m=+273.513233957" watchObservedRunningTime="2026-09-11 18:20:05.287140285 +0000 UTC m=+273.513816626" Sep 11 18:20:14.689410 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:14.689354 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:14.689821 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:14.689439 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:14.694299 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:14.694274 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:15.308369 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:15.308342 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:20:15.355232 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:15.355200 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbd7656bf-jrw2n"] Sep 11 18:20:15.464676 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:15.464648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:15.569212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:15.569117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:15.666079 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:15.666048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:20.102154 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:20.102115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:32.173835 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:32.173807 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:20:32.178786 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:32.178762 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:20:32.184389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:32.184363 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 18:20:40.377887 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.377819 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fbd7656bf-jrw2n" podUID="9b0b5cae-637c-402c-ab3c-09f0c8e94387" containerName="console" containerID="cri-o://729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61" gracePeriod=15 Sep 11 18:20:40.618641 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.618616 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:20:40.809116 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809081 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqxvz\" (UniqueName: \"kubernetes.io/projected/9b0b5cae-637c-402c-ab3c-09f0c8e94387-kube-api-access-sqxvz\") pod \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " Sep 11 18:20:40.809284 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809126 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-trusted-ca-bundle\") pod \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " Sep 11 18:20:40.809284 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809196 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-serving-cert\") pod \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " Sep 11 18:20:40.809284 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809230 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-config\") pod \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " Sep 11 18:20:40.809284 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809263 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-service-ca\") pod \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " Sep 11 18:20:40.809505 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809303 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-oauth-serving-cert\") pod \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " Sep 11 18:20:40.809505 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809333 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-oauth-config\") pod \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\" (UID: \"9b0b5cae-637c-402c-ab3c-09f0c8e94387\") " Sep 11 18:20:40.809660 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809607 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9b0b5cae-637c-402c-ab3c-09f0c8e94387" (UID: "9b0b5cae-637c-402c-ab3c-09f0c8e94387"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:20:40.809771 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809675 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-config" (OuterVolumeSpecName: "console-config") pod "9b0b5cae-637c-402c-ab3c-09f0c8e94387" (UID: "9b0b5cae-637c-402c-ab3c-09f0c8e94387"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:20:40.809771 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809689 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b0b5cae-637c-402c-ab3c-09f0c8e94387" (UID: "9b0b5cae-637c-402c-ab3c-09f0c8e94387"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:20:40.809771 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.809678 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b0b5cae-637c-402c-ab3c-09f0c8e94387" (UID: "9b0b5cae-637c-402c-ab3c-09f0c8e94387"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:20:40.811277 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.811251 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b0b5cae-637c-402c-ab3c-09f0c8e94387" (UID: "9b0b5cae-637c-402c-ab3c-09f0c8e94387"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:20:40.811391 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.811330 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0b5cae-637c-402c-ab3c-09f0c8e94387-kube-api-access-sqxvz" (OuterVolumeSpecName: "kube-api-access-sqxvz") pod "9b0b5cae-637c-402c-ab3c-09f0c8e94387" (UID: "9b0b5cae-637c-402c-ab3c-09f0c8e94387"). InnerVolumeSpecName "kube-api-access-sqxvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:20:40.811442 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.811419 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b0b5cae-637c-402c-ab3c-09f0c8e94387" (UID: "9b0b5cae-637c-402c-ab3c-09f0c8e94387"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:20:40.915035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.910998 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:20:40.915035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.911050 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:20:40.915035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.911066 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:20:40.915035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.911081 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-oauth-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:20:40.915035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.911102 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0b5cae-637c-402c-ab3c-09f0c8e94387-console-oauth-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:20:40.915035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.911116 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxvz\" (UniqueName: \"kubernetes.io/projected/9b0b5cae-637c-402c-ab3c-09f0c8e94387-kube-api-access-sqxvz\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:20:40.915035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:40.911131 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0b5cae-637c-402c-ab3c-09f0c8e94387-trusted-ca-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:20:41.404503 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:41.404464 2579 generic.go:356] "Generic (PLEG): container finished" podID="9b0b5cae-637c-402c-ab3c-09f0c8e94387" containerID="729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61" exitCode=0 Sep 11 18:20:41.404912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:41.404524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbd7656bf-jrw2n" event={"ID":"9b0b5cae-637c-402c-ab3c-09f0c8e94387","Type":"ContainerDied","Data":"729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61"} Sep 11 18:20:41.404912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:41.404529 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbd7656bf-jrw2n" Sep 11 18:20:41.404912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:41.404566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbd7656bf-jrw2n" event={"ID":"9b0b5cae-637c-402c-ab3c-09f0c8e94387","Type":"ContainerDied","Data":"384da626590cb504b3e9f43548d18c618bae79bf68819edbfc730c0bf3ff3d1f"} Sep 11 18:20:41.404912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:41.404585 2579 scope.go:117] "RemoveContainer" containerID="729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61" Sep 11 18:20:41.413547 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:41.413530 2579 scope.go:117] "RemoveContainer" containerID="729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61" Sep 11 18:20:41.413822 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:20:41.413804 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61\": container with ID starting with 729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61 not found: ID does not exist" containerID="729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61" Sep 11 18:20:41.413873 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:41.413829 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61"} err="failed to get container status \"729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61\": rpc error: code = NotFound desc = could not find container \"729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61\": container with ID starting with 729ce4887416673f6cf0e32d2d4c594569d2b3c4325fc3dabaa277f476622f61 not found: ID does not exist" Sep 11 18:20:41.426920 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:41.426898 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbd7656bf-jrw2n"] Sep 11 18:20:41.431394 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:41.431375 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fbd7656bf-jrw2n"] Sep 11 18:20:42.228262 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:42.228227 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0b5cae-637c-402c-ab3c-09f0c8e94387" path="/var/lib/kubelet/pods/9b0b5cae-637c-402c-ab3c-09f0c8e94387/volumes" Sep 11 18:20:44.953242 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:44.953202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:45.010513 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:20:45.010478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:10.979191 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:10.979156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:11.049091 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:11.049054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:18.822477 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.822445 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b98898d58-2fd8r"] Sep 11 18:21:18.822960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.822766 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0b5cae-637c-402c-ab3c-09f0c8e94387" containerName="console" Sep 11 18:21:18.822960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.822779 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0b5cae-637c-402c-ab3c-09f0c8e94387" containerName="console" Sep 11 18:21:18.822960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.822860 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0b5cae-637c-402c-ab3c-09f0c8e94387" containerName="console" Sep 11 18:21:18.826073 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.826054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.842955 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.842928 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b98898d58-2fd8r"] Sep 11 18:21:18.890900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.890868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-trusted-ca-bundle\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.890900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.890906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-console-config\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.891097 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.890930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-serving-cert\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.891097 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.891004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-service-ca\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.891097 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.891040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-oauth-config\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.891097 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.891055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-oauth-serving-cert\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.891233 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.891099 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g6pk\" (UniqueName: \"kubernetes.io/projected/45257a96-959d-4e32-a548-ab1577f9f7e1-kube-api-access-4g6pk\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.904604 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.904568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:18.972275 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.972242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:18.992287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.992265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-oauth-config\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.992370 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.992298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-oauth-serving-cert\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.992370 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.992324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g6pk\" (UniqueName: \"kubernetes.io/projected/45257a96-959d-4e32-a548-ab1577f9f7e1-kube-api-access-4g6pk\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.992370 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.992363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-trusted-ca-bundle\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.992508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.992386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-console-config\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.992508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.992410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-serving-cert\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.992508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.992438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-service-ca\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.993245 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.993219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-oauth-serving-cert\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.993352 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.993220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-service-ca\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.993352 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.993288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-trusted-ca-bundle\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.993430 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.993377 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-console-config\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.994930 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.994914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-oauth-config\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:18.994980 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:18.994945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-serving-cert\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:19.001125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:19.001101 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g6pk\" (UniqueName: \"kubernetes.io/projected/45257a96-959d-4e32-a548-ab1577f9f7e1-kube-api-access-4g6pk\") pod \"console-b98898d58-2fd8r\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:19.134584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:19.134493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:19.414410 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:19.414384 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b98898d58-2fd8r"] Sep 11 18:21:19.416865 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:21:19.416837 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45257a96_959d_4e32_a548_ab1577f9f7e1.slice/crio-594ce418023f09cc81b87a529be718dec51a8d87392f27f3fc0ea20e34701872 WatchSource:0}: Error finding container 594ce418023f09cc81b87a529be718dec51a8d87392f27f3fc0ea20e34701872: Status 404 returned error can't find the container with id 594ce418023f09cc81b87a529be718dec51a8d87392f27f3fc0ea20e34701872 Sep 11 18:21:19.551845 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:19.551813 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b98898d58-2fd8r" event={"ID":"45257a96-959d-4e32-a548-ab1577f9f7e1","Type":"ContainerStarted","Data":"5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891"} Sep 11 18:21:19.552022 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:19.551864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b98898d58-2fd8r" event={"ID":"45257a96-959d-4e32-a548-ab1577f9f7e1","Type":"ContainerStarted","Data":"594ce418023f09cc81b87a529be718dec51a8d87392f27f3fc0ea20e34701872"} Sep 11 18:21:19.574834 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:19.574781 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b98898d58-2fd8r" podStartSLOduration=1.574763771 podStartE2EDuration="1.574763771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:19.573983484 +0000 UTC m=+347.800659850" watchObservedRunningTime="2026-09-11 18:21:19.574763771 +0000 UTC m=+347.801440206" Sep 11 18:21:22.960740 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:22.960685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:23.037112 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:23.037078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:29.135237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:29.135129 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:29.135237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:29.135222 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:29.140042 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:29.140017 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:29.593648 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:29.593622 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:21:29.656486 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:29.656456 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4fff9697-spwr2"] Sep 11 18:21:29.774859 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:29.774825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:29.874156 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:29.874069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:29.948293 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:29.948263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:36.530127 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:36.530086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:36.632013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:36.631964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:52.286154 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:52.286110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:54.676384 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:54.676342 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c4fff9697-spwr2" podUID="af68fbed-4f8e-40dc-8f68-d5570770e214" containerName="console" containerID="cri-o://e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04" gracePeriod=15 Sep 11 18:21:54.915584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:54.915561 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:21:55.086890 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.086858 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-oauth-config\") pod \"af68fbed-4f8e-40dc-8f68-d5570770e214\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " Sep 11 18:21:55.087103 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.086903 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-trusted-ca-bundle\") pod \"af68fbed-4f8e-40dc-8f68-d5570770e214\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " Sep 11 18:21:55.087103 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.086923 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-console-config\") pod \"af68fbed-4f8e-40dc-8f68-d5570770e214\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " Sep 11 18:21:55.087103 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.086951 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-service-ca\") pod \"af68fbed-4f8e-40dc-8f68-d5570770e214\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " Sep 11 18:21:55.087103 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.086986 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9q89\" (UniqueName: \"kubernetes.io/projected/af68fbed-4f8e-40dc-8f68-d5570770e214-kube-api-access-v9q89\") pod \"af68fbed-4f8e-40dc-8f68-d5570770e214\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " Sep 11 18:21:55.087103 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.087010 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-serving-cert\") pod \"af68fbed-4f8e-40dc-8f68-d5570770e214\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " Sep 11 18:21:55.087103 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.087045 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-oauth-serving-cert\") pod \"af68fbed-4f8e-40dc-8f68-d5570770e214\" (UID: \"af68fbed-4f8e-40dc-8f68-d5570770e214\") " Sep 11 18:21:55.087451 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.087402 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "af68fbed-4f8e-40dc-8f68-d5570770e214" (UID: "af68fbed-4f8e-40dc-8f68-d5570770e214"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:55.087492 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.087461 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-console-config" (OuterVolumeSpecName: "console-config") pod "af68fbed-4f8e-40dc-8f68-d5570770e214" (UID: "af68fbed-4f8e-40dc-8f68-d5570770e214"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:55.087777 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.087755 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "af68fbed-4f8e-40dc-8f68-d5570770e214" (UID: "af68fbed-4f8e-40dc-8f68-d5570770e214"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:55.087885 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.087809 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-service-ca" (OuterVolumeSpecName: "service-ca") pod "af68fbed-4f8e-40dc-8f68-d5570770e214" (UID: "af68fbed-4f8e-40dc-8f68-d5570770e214"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:55.089250 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.089216 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "af68fbed-4f8e-40dc-8f68-d5570770e214" (UID: "af68fbed-4f8e-40dc-8f68-d5570770e214"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:21:55.089359 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.089253 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "af68fbed-4f8e-40dc-8f68-d5570770e214" (UID: "af68fbed-4f8e-40dc-8f68-d5570770e214"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:21:55.089359 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.089296 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af68fbed-4f8e-40dc-8f68-d5570770e214-kube-api-access-v9q89" (OuterVolumeSpecName: "kube-api-access-v9q89") pod "af68fbed-4f8e-40dc-8f68-d5570770e214" (UID: "af68fbed-4f8e-40dc-8f68-d5570770e214"). InnerVolumeSpecName "kube-api-access-v9q89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:55.188045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.188010 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-oauth-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:21:55.188045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.188038 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-trusted-ca-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:21:55.188045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.188048 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-console-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:21:55.188045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.188057 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:21:55.188305 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.188065 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9q89\" (UniqueName: \"kubernetes.io/projected/af68fbed-4f8e-40dc-8f68-d5570770e214-kube-api-access-v9q89\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:21:55.188305 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.188076 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af68fbed-4f8e-40dc-8f68-d5570770e214-console-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:21:55.188305 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.188084 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af68fbed-4f8e-40dc-8f68-d5570770e214-oauth-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:21:55.684719 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.684668 2579 generic.go:356] "Generic (PLEG): container finished" podID="af68fbed-4f8e-40dc-8f68-d5570770e214" containerID="e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04" exitCode=0 Sep 11 18:21:55.685090 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.684727 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4fff9697-spwr2" Sep 11 18:21:55.685090 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.684736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4fff9697-spwr2" event={"ID":"af68fbed-4f8e-40dc-8f68-d5570770e214","Type":"ContainerDied","Data":"e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04"} Sep 11 18:21:55.685090 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.684780 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4fff9697-spwr2" event={"ID":"af68fbed-4f8e-40dc-8f68-d5570770e214","Type":"ContainerDied","Data":"f00e8779f46969afa92f7033999018a35518fd802cf651481e2ac220f41ae4d2"} Sep 11 18:21:55.685090 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.684799 2579 scope.go:117] "RemoveContainer" containerID="e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04" Sep 11 18:21:55.695960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.695936 2579 scope.go:117] "RemoveContainer" containerID="e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04" Sep 11 18:21:55.696514 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:21:55.696486 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04\": container with ID starting with e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04 not found: ID does not exist" containerID="e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04" Sep 11 18:21:55.696596 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.696523 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04"} err="failed to get container status \"e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04\": rpc error: code = NotFound desc = could not find container \"e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04\": container with ID starting with e774dbafc3b854c847040dcb947fa159235d2f53ffd85f8fe301116b6ff43d04 not found: ID does not exist" Sep 11 18:21:55.710182 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.710161 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4fff9697-spwr2"] Sep 11 18:21:55.714951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.714928 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4fff9697-spwr2"] Sep 11 18:21:55.976522 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:55.976429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:56.078720 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:56.078677 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:56.227644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:56.227567 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af68fbed-4f8e-40dc-8f68-d5570770e214" path="/var/lib/kubelet/pods/af68fbed-4f8e-40dc-8f68-d5570770e214/volumes" Sep 11 18:21:57.051195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.051158 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq"] Sep 11 18:21:57.051557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.051441 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af68fbed-4f8e-40dc-8f68-d5570770e214" containerName="console" Sep 11 18:21:57.051557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.051452 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af68fbed-4f8e-40dc-8f68-d5570770e214" containerName="console" Sep 11 18:21:57.051557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.051502 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af68fbed-4f8e-40dc-8f68-d5570770e214" containerName="console" Sep 11 18:21:57.060826 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.060794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.064845 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.064760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lgxhw\"" Sep 11 18:21:57.065087 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.065061 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq"] Sep 11 18:21:57.065354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.065328 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 18:21:57.065761 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.065694 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 18:21:57.204570 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.204527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.204570 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.204570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkvws\" (UniqueName: \"kubernetes.io/projected/78208883-f6bb-4736-b11a-f7777d4c298d-kube-api-access-mkvws\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.204816 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.204595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.305698 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.305629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.305698 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.305667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkvws\" (UniqueName: \"kubernetes.io/projected/78208883-f6bb-4736-b11a-f7777d4c298d-kube-api-access-mkvws\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.305866 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.305696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.306111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.306091 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.306149 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.306106 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.316064 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.316033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkvws\" (UniqueName: \"kubernetes.io/projected/78208883-f6bb-4736-b11a-f7777d4c298d-kube-api-access-mkvws\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.371738 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.371692 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:21:57.642477 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.642401 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq"] Sep 11 18:21:57.645071 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:21:57.645033 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78208883_f6bb_4736_b11a_f7777d4c298d.slice/crio-0e201b41fdc7b2865fa30315528bc52abacec86d6cf99a785347c627e563c4b2 WatchSource:0}: Error finding container 0e201b41fdc7b2865fa30315528bc52abacec86d6cf99a785347c627e563c4b2: Status 404 returned error can't find the container with id 0e201b41fdc7b2865fa30315528bc52abacec86d6cf99a785347c627e563c4b2 Sep 11 18:21:57.646898 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.646882 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:21:57.693541 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:21:57.693505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" event={"ID":"78208883-f6bb-4736-b11a-f7777d4c298d","Type":"ContainerStarted","Data":"0e201b41fdc7b2865fa30315528bc52abacec86d6cf99a785347c627e563c4b2"} Sep 11 18:22:04.723601 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:04.723566 2579 generic.go:356] "Generic (PLEG): container finished" podID="78208883-f6bb-4736-b11a-f7777d4c298d" containerID="1f943370f1d05bf0c4a957cbf8efb656eb0ddc6fc6f9ce6f32a2286fc0119230" exitCode=0 Sep 11 18:22:04.724012 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:04.723627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" event={"ID":"78208883-f6bb-4736-b11a-f7777d4c298d","Type":"ContainerDied","Data":"1f943370f1d05bf0c4a957cbf8efb656eb0ddc6fc6f9ce6f32a2286fc0119230"} Sep 11 18:22:07.737938 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:07.737901 2579 generic.go:356] "Generic (PLEG): container finished" podID="78208883-f6bb-4736-b11a-f7777d4c298d" containerID="1dbb96315596f6f8f454b4f4254dd215b5be7d5286f8c51f8da6d060eaa4b610" exitCode=0 Sep 11 18:22:07.738312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:07.737986 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" event={"ID":"78208883-f6bb-4736-b11a-f7777d4c298d","Type":"ContainerDied","Data":"1dbb96315596f6f8f454b4f4254dd215b5be7d5286f8c51f8da6d060eaa4b610"} Sep 11 18:22:15.771207 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:15.771167 2579 generic.go:356] "Generic (PLEG): container finished" podID="78208883-f6bb-4736-b11a-f7777d4c298d" containerID="9bf15054ccfb022445ace54f6af58eacb71b9bdde9be54dde8067658fdc475c8" exitCode=0 Sep 11 18:22:15.771674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:15.771260 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" event={"ID":"78208883-f6bb-4736-b11a-f7777d4c298d","Type":"ContainerDied","Data":"9bf15054ccfb022445ace54f6af58eacb71b9bdde9be54dde8067658fdc475c8"} Sep 11 18:22:16.943580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:16.943545 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:22:17.075513 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.075427 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkvws\" (UniqueName: \"kubernetes.io/projected/78208883-f6bb-4736-b11a-f7777d4c298d-kube-api-access-mkvws\") pod \"78208883-f6bb-4736-b11a-f7777d4c298d\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " Sep 11 18:22:17.075513 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.075490 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-bundle\") pod \"78208883-f6bb-4736-b11a-f7777d4c298d\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " Sep 11 18:22:17.075698 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.075528 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-util\") pod \"78208883-f6bb-4736-b11a-f7777d4c298d\" (UID: \"78208883-f6bb-4736-b11a-f7777d4c298d\") " Sep 11 18:22:17.076208 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.076174 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-bundle" (OuterVolumeSpecName: "bundle") pod "78208883-f6bb-4736-b11a-f7777d4c298d" (UID: "78208883-f6bb-4736-b11a-f7777d4c298d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:22:17.077679 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.077650 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78208883-f6bb-4736-b11a-f7777d4c298d-kube-api-access-mkvws" (OuterVolumeSpecName: "kube-api-access-mkvws") pod "78208883-f6bb-4736-b11a-f7777d4c298d" (UID: "78208883-f6bb-4736-b11a-f7777d4c298d"). InnerVolumeSpecName "kube-api-access-mkvws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:17.079896 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.079870 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-util" (OuterVolumeSpecName: "util") pod "78208883-f6bb-4736-b11a-f7777d4c298d" (UID: "78208883-f6bb-4736-b11a-f7777d4c298d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:22:17.176658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.176618 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:22:17.176658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.176651 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78208883-f6bb-4736-b11a-f7777d4c298d-util\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:22:17.176658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.176662 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkvws\" (UniqueName: \"kubernetes.io/projected/78208883-f6bb-4736-b11a-f7777d4c298d-kube-api-access-mkvws\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:22:17.781511 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.781477 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" Sep 11 18:22:17.781688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.781481 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdvjq" event={"ID":"78208883-f6bb-4736-b11a-f7777d4c298d","Type":"ContainerDied","Data":"0e201b41fdc7b2865fa30315528bc52abacec86d6cf99a785347c627e563c4b2"} Sep 11 18:22:17.781688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:17.781603 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e201b41fdc7b2865fa30315528bc52abacec86d6cf99a785347c627e563c4b2" Sep 11 18:22:22.895681 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:22.895644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:24.231784 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.231749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:24.777497 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.777463 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv"] Sep 11 18:22:24.777782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.777769 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78208883-f6bb-4736-b11a-f7777d4c298d" containerName="util" Sep 11 18:22:24.777832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.777784 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="78208883-f6bb-4736-b11a-f7777d4c298d" containerName="util" Sep 11 18:22:24.777832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.777794 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78208883-f6bb-4736-b11a-f7777d4c298d" containerName="extract" Sep 11 18:22:24.777832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.777800 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="78208883-f6bb-4736-b11a-f7777d4c298d" containerName="extract" Sep 11 18:22:24.777832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.777810 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78208883-f6bb-4736-b11a-f7777d4c298d" containerName="pull" Sep 11 18:22:24.777832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.777816 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="78208883-f6bb-4736-b11a-f7777d4c298d" containerName="pull" Sep 11 18:22:24.777981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.777862 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="78208883-f6bb-4736-b11a-f7777d4c298d" containerName="extract" Sep 11 18:22:24.779725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.779693 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" Sep 11 18:22:24.782320 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.782298 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 18:22:24.782895 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.782877 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:22:24.783389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.783369 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dqmhr\"" Sep 11 18:22:24.797037 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.797016 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv"] Sep 11 18:22:24.838556 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.838523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11928c35-5585-44ca-8446-ac0ce1ac1603-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lg6cv\" (UID: \"11928c35-5585-44ca-8446-ac0ce1ac1603\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" Sep 11 18:22:24.838737 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.838569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6z6l\" (UniqueName: \"kubernetes.io/projected/11928c35-5585-44ca-8446-ac0ce1ac1603-kube-api-access-n6z6l\") pod \"cert-manager-operator-controller-manager-55c444bf86-lg6cv\" (UID: \"11928c35-5585-44ca-8446-ac0ce1ac1603\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" Sep 11 18:22:24.939687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.939647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z6l\" (UniqueName: \"kubernetes.io/projected/11928c35-5585-44ca-8446-ac0ce1ac1603-kube-api-access-n6z6l\") pod \"cert-manager-operator-controller-manager-55c444bf86-lg6cv\" (UID: \"11928c35-5585-44ca-8446-ac0ce1ac1603\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" Sep 11 18:22:24.939879 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.939760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11928c35-5585-44ca-8446-ac0ce1ac1603-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lg6cv\" (UID: \"11928c35-5585-44ca-8446-ac0ce1ac1603\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" Sep 11 18:22:24.940095 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.940080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11928c35-5585-44ca-8446-ac0ce1ac1603-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lg6cv\" (UID: \"11928c35-5585-44ca-8446-ac0ce1ac1603\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" Sep 11 18:22:24.948570 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:24.948547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6z6l\" (UniqueName: \"kubernetes.io/projected/11928c35-5585-44ca-8446-ac0ce1ac1603-kube-api-access-n6z6l\") pod \"cert-manager-operator-controller-manager-55c444bf86-lg6cv\" (UID: \"11928c35-5585-44ca-8446-ac0ce1ac1603\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" Sep 11 18:22:25.088838 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:25.088762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" Sep 11 18:22:25.362186 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:25.362161 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv"] Sep 11 18:22:25.364845 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:22:25.364819 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11928c35_5585_44ca_8446_ac0ce1ac1603.slice/crio-e12a510e9ae676cda051fd2c85e69d8d6d7ee78c6aa418d0d9aca4084dd97104 WatchSource:0}: Error finding container e12a510e9ae676cda051fd2c85e69d8d6d7ee78c6aa418d0d9aca4084dd97104: Status 404 returned error can't find the container with id e12a510e9ae676cda051fd2c85e69d8d6d7ee78c6aa418d0d9aca4084dd97104 Sep 11 18:22:25.813390 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:25.813350 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" event={"ID":"11928c35-5585-44ca-8446-ac0ce1ac1603","Type":"ContainerStarted","Data":"e12a510e9ae676cda051fd2c85e69d8d6d7ee78c6aa418d0d9aca4084dd97104"} Sep 11 18:22:26.479188 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:26.479159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:26.580727 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:26.580663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:29.833651 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:29.833611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" event={"ID":"11928c35-5585-44ca-8446-ac0ce1ac1603","Type":"ContainerStarted","Data":"7df10d23a81f9d6cf3b3129a1a4bd14ac2d46272ce64d1098509118c8d81daf9"} Sep 11 18:22:29.861231 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:29.861184 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lg6cv" podStartSLOduration=1.7339183440000001 podStartE2EDuration="5.86116745s" totalImagesPullingTime="4.127249106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:29.860235053 +0000 UTC m=+418.086911401" watchObservedRunningTime="2026-09-11 18:22:29.86116745 +0000 UTC m=+418.087843790" Sep 11 18:22:30.254412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.254376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:30.430658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.430620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:30.721613 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.721526 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fsm6k"] Sep 11 18:22:30.724999 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.724980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:30.727808 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.727791 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tr8k5\"" Sep 11 18:22:30.728733 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.728696 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 18:22:30.728811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.728755 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 18:22:30.742688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.742659 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fsm6k"] Sep 11 18:22:30.786997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.786959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv5q4\" (UniqueName: \"kubernetes.io/projected/d253a9c1-97de-4b59-b086-fd9ff7ef5a06-kube-api-access-nv5q4\") pod \"cert-manager-webhook-6486ffd997-fsm6k\" (UID: \"d253a9c1-97de-4b59-b086-fd9ff7ef5a06\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:30.787153 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.787011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d253a9c1-97de-4b59-b086-fd9ff7ef5a06-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fsm6k\" (UID: \"d253a9c1-97de-4b59-b086-fd9ff7ef5a06\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:30.887789 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.887751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv5q4\" (UniqueName: \"kubernetes.io/projected/d253a9c1-97de-4b59-b086-fd9ff7ef5a06-kube-api-access-nv5q4\") pod \"cert-manager-webhook-6486ffd997-fsm6k\" (UID: \"d253a9c1-97de-4b59-b086-fd9ff7ef5a06\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:30.888249 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.887804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d253a9c1-97de-4b59-b086-fd9ff7ef5a06-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fsm6k\" (UID: \"d253a9c1-97de-4b59-b086-fd9ff7ef5a06\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:30.910646 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.910615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d253a9c1-97de-4b59-b086-fd9ff7ef5a06-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fsm6k\" (UID: \"d253a9c1-97de-4b59-b086-fd9ff7ef5a06\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:30.911468 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:30.911450 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv5q4\" (UniqueName: \"kubernetes.io/projected/d253a9c1-97de-4b59-b086-fd9ff7ef5a06-kube-api-access-nv5q4\") pod \"cert-manager-webhook-6486ffd997-fsm6k\" (UID: \"d253a9c1-97de-4b59-b086-fd9ff7ef5a06\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:31.049185 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:31.049144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:31.323874 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:31.323848 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fsm6k"] Sep 11 18:22:31.326467 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:22:31.326441 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd253a9c1_97de_4b59_b086_fd9ff7ef5a06.slice/crio-6db0a253b676d07d48e6332c2100e2e171d9f3c90fe86c534014a46421bd27dd WatchSource:0}: Error finding container 6db0a253b676d07d48e6332c2100e2e171d9f3c90fe86c534014a46421bd27dd: Status 404 returned error can't find the container with id 6db0a253b676d07d48e6332c2100e2e171d9f3c90fe86c534014a46421bd27dd Sep 11 18:22:31.844205 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:31.844165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" event={"ID":"d253a9c1-97de-4b59-b086-fd9ff7ef5a06","Type":"ContainerStarted","Data":"6db0a253b676d07d48e6332c2100e2e171d9f3c90fe86c534014a46421bd27dd"} Sep 11 18:22:31.907863 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:31.907824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:31.974758 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:31.974696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:32.642296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:32.642256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:32.855469 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:32.855416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:34.860460 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:34.860428 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" event={"ID":"d253a9c1-97de-4b59-b086-fd9ff7ef5a06","Type":"ContainerStarted","Data":"f3101e94dc63912316ed46ce02372b73dad93d8645a34699f3a19e69ae9d9c4c"} Sep 11 18:22:34.860915 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:34.860525 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:34.879404 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:34.879353 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" podStartSLOduration=2.113926724 podStartE2EDuration="4.879333654s" totalImagesPullingTime="2.76540693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:34.879000694 +0000 UTC m=+423.105677037" watchObservedRunningTime="2026-09-11 18:22:34.879333654 +0000 UTC m=+423.106009988" Sep 11 18:22:38.416475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.416439 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5"] Sep 11 18:22:38.419647 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.419632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" Sep 11 18:22:38.423184 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.423155 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4dm4c\"" Sep 11 18:22:38.444121 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.444096 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5"] Sep 11 18:22:38.549606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.549568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8l6\" (UniqueName: \"kubernetes.io/projected/1400523d-2efa-400a-b573-f0409613d7b5-kube-api-access-tj8l6\") pod \"cert-manager-cainjector-559cdd8c87-lgrj5\" (UID: \"1400523d-2efa-400a-b573-f0409613d7b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" Sep 11 18:22:38.549606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.549607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1400523d-2efa-400a-b573-f0409613d7b5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lgrj5\" (UID: \"1400523d-2efa-400a-b573-f0409613d7b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" Sep 11 18:22:38.650722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.650667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8l6\" (UniqueName: \"kubernetes.io/projected/1400523d-2efa-400a-b573-f0409613d7b5-kube-api-access-tj8l6\") pod \"cert-manager-cainjector-559cdd8c87-lgrj5\" (UID: \"1400523d-2efa-400a-b573-f0409613d7b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" Sep 11 18:22:38.650722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.650718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1400523d-2efa-400a-b573-f0409613d7b5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lgrj5\" (UID: \"1400523d-2efa-400a-b573-f0409613d7b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" Sep 11 18:22:38.660297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.660266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8l6\" (UniqueName: \"kubernetes.io/projected/1400523d-2efa-400a-b573-f0409613d7b5-kube-api-access-tj8l6\") pod \"cert-manager-cainjector-559cdd8c87-lgrj5\" (UID: \"1400523d-2efa-400a-b573-f0409613d7b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" Sep 11 18:22:38.660574 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.660552 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1400523d-2efa-400a-b573-f0409613d7b5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lgrj5\" (UID: \"1400523d-2efa-400a-b573-f0409613d7b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" Sep 11 18:22:38.731194 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:38.731092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" Sep 11 18:22:39.001392 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:39.001328 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5"] Sep 11 18:22:39.004093 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:22:39.004060 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1400523d_2efa_400a_b573_f0409613d7b5.slice/crio-95757de858034459edf1a9203b72a59f687f21d8a615487d79c83b4e35d44021 WatchSource:0}: Error finding container 95757de858034459edf1a9203b72a59f687f21d8a615487d79c83b4e35d44021: Status 404 returned error can't find the container with id 95757de858034459edf1a9203b72a59f687f21d8a615487d79c83b4e35d44021 Sep 11 18:22:39.786027 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:39.785993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:39.860689 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:39.860651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:39.883651 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:39.883614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" event={"ID":"1400523d-2efa-400a-b573-f0409613d7b5","Type":"ContainerStarted","Data":"6aca8f8e657e8ba7bd10ec00829fd9f2ffeae2d67eeafd9bdd06d1615368af7a"} Sep 11 18:22:39.883857 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:39.883660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" event={"ID":"1400523d-2efa-400a-b573-f0409613d7b5","Type":"ContainerStarted","Data":"95757de858034459edf1a9203b72a59f687f21d8a615487d79c83b4e35d44021"} Sep 11 18:22:39.921172 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:39.921127 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lgrj5" podStartSLOduration=1.92111224 podStartE2EDuration="1.92111224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:39.920426911 +0000 UTC m=+428.147103251" watchObservedRunningTime="2026-09-11 18:22:39.92111224 +0000 UTC m=+428.147788579" Sep 11 18:22:40.464252 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:40.464215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:40.867771 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:40.867740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fsm6k" Sep 11 18:22:41.504484 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.504452 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mq959"] Sep 11 18:22:41.507527 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.507509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mq959" Sep 11 18:22:41.510066 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.510050 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2l8wn\"" Sep 11 18:22:41.520469 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.520443 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mq959"] Sep 11 18:22:41.674917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.674882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cdd1690-1e05-4e92-965a-3cc0f107effc-bound-sa-token\") pod \"cert-manager-55b4d954d5-mq959\" (UID: \"1cdd1690-1e05-4e92-965a-3cc0f107effc\") " pod="cert-manager/cert-manager-55b4d954d5-mq959" Sep 11 18:22:41.675086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.674947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29s8\" (UniqueName: \"kubernetes.io/projected/1cdd1690-1e05-4e92-965a-3cc0f107effc-kube-api-access-t29s8\") pod \"cert-manager-55b4d954d5-mq959\" (UID: \"1cdd1690-1e05-4e92-965a-3cc0f107effc\") " pod="cert-manager/cert-manager-55b4d954d5-mq959" Sep 11 18:22:41.775899 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.775813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29s8\" (UniqueName: \"kubernetes.io/projected/1cdd1690-1e05-4e92-965a-3cc0f107effc-kube-api-access-t29s8\") pod \"cert-manager-55b4d954d5-mq959\" (UID: \"1cdd1690-1e05-4e92-965a-3cc0f107effc\") " pod="cert-manager/cert-manager-55b4d954d5-mq959" Sep 11 18:22:41.775899 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.775875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cdd1690-1e05-4e92-965a-3cc0f107effc-bound-sa-token\") pod \"cert-manager-55b4d954d5-mq959\" (UID: \"1cdd1690-1e05-4e92-965a-3cc0f107effc\") " pod="cert-manager/cert-manager-55b4d954d5-mq959" Sep 11 18:22:41.784792 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.784765 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cdd1690-1e05-4e92-965a-3cc0f107effc-bound-sa-token\") pod \"cert-manager-55b4d954d5-mq959\" (UID: \"1cdd1690-1e05-4e92-965a-3cc0f107effc\") " pod="cert-manager/cert-manager-55b4d954d5-mq959" Sep 11 18:22:41.784903 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.784837 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29s8\" (UniqueName: \"kubernetes.io/projected/1cdd1690-1e05-4e92-965a-3cc0f107effc-kube-api-access-t29s8\") pod \"cert-manager-55b4d954d5-mq959\" (UID: \"1cdd1690-1e05-4e92-965a-3cc0f107effc\") " pod="cert-manager/cert-manager-55b4d954d5-mq959" Sep 11 18:22:41.816730 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:41.816677 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mq959" Sep 11 18:22:42.094132 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:42.094106 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mq959"] Sep 11 18:22:42.096287 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:22:42.096261 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cdd1690_1e05_4e92_965a_3cc0f107effc.slice/crio-afed6524c93abc01bcd5b76ead4f2167105be42dd17f5a03d64f2adc9502d746 WatchSource:0}: Error finding container afed6524c93abc01bcd5b76ead4f2167105be42dd17f5a03d64f2adc9502d746: Status 404 returned error can't find the container with id afed6524c93abc01bcd5b76ead4f2167105be42dd17f5a03d64f2adc9502d746 Sep 11 18:22:42.899390 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:42.899350 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mq959" event={"ID":"1cdd1690-1e05-4e92-965a-3cc0f107effc","Type":"ContainerStarted","Data":"23842cfea281625c93959193fb51cdcc9cd1e204b3a1c67acbaf586967568f9a"} Sep 11 18:22:42.899560 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:42.899399 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mq959" event={"ID":"1cdd1690-1e05-4e92-965a-3cc0f107effc","Type":"ContainerStarted","Data":"afed6524c93abc01bcd5b76ead4f2167105be42dd17f5a03d64f2adc9502d746"} Sep 11 18:22:42.919327 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:42.919280 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mq959" podStartSLOduration=1.919265421 podStartE2EDuration="1.919265421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:42.917527808 +0000 UTC m=+431.144204147" watchObservedRunningTime="2026-09-11 18:22:42.919265421 +0000 UTC m=+431.145941761" Sep 11 18:22:54.238262 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:54.238223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:56.025574 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:22:56.025534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:00.824725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:00.824662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:00.925688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:00.925650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:02.377623 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.377589 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2"] Sep 11 18:23:02.380995 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.380976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.383541 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.383524 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lgxhw\"" Sep 11 18:23:02.383615 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.383576 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 18:23:02.384583 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.384559 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 18:23:02.392475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.392453 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2"] Sep 11 18:23:02.444079 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.444051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.444250 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.444107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.444250 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.444130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbrhn\" (UniqueName: \"kubernetes.io/projected/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-kube-api-access-pbrhn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.544481 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.544447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.544655 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.544495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbrhn\" (UniqueName: \"kubernetes.io/projected/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-kube-api-access-pbrhn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.544655 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.544540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.544882 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.544851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.544953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.544901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.555684 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.555654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbrhn\" (UniqueName: \"kubernetes.io/projected/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-kube-api-access-pbrhn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.673247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.673155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:02.690902 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.690873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:02.976247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:02.976171 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2"] Sep 11 18:23:02.980510 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:23:02.980475 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43a6cc6_f2f7_4b8d_9ddd_34088ecb0c9d.slice/crio-5ea30c80c6b50c13f1a6dfad5581747c9d4448ddb82e59b5abcdb8dde740db12 WatchSource:0}: Error finding container 5ea30c80c6b50c13f1a6dfad5581747c9d4448ddb82e59b5abcdb8dde740db12: Status 404 returned error can't find the container with id 5ea30c80c6b50c13f1a6dfad5581747c9d4448ddb82e59b5abcdb8dde740db12 Sep 11 18:23:03.985944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:03.985857 2579 generic.go:356] "Generic (PLEG): container finished" podID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerID="25379afeac226424643144e6c141d6e95cb39947b7c87e1c73a10a9ad25edd01" exitCode=0 Sep 11 18:23:03.986298 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:03.985951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" event={"ID":"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d","Type":"ContainerDied","Data":"25379afeac226424643144e6c141d6e95cb39947b7c87e1c73a10a9ad25edd01"} Sep 11 18:23:03.986298 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:03.986001 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" event={"ID":"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d","Type":"ContainerStarted","Data":"5ea30c80c6b50c13f1a6dfad5581747c9d4448ddb82e59b5abcdb8dde740db12"} Sep 11 18:23:04.993166 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:04.993135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" event={"ID":"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d","Type":"ContainerStarted","Data":"b1cc5b645d3a16c952a03a35d136cba2de875d833679108c372e92c2651bd2a1"} Sep 11 18:23:05.999658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:05.999624 2579 generic.go:356] "Generic (PLEG): container finished" podID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerID="b1cc5b645d3a16c952a03a35d136cba2de875d833679108c372e92c2651bd2a1" exitCode=0 Sep 11 18:23:06.000049 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:05.999686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" event={"ID":"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d","Type":"ContainerDied","Data":"b1cc5b645d3a16c952a03a35d136cba2de875d833679108c372e92c2651bd2a1"} Sep 11 18:23:07.006226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:07.006188 2579 generic.go:356] "Generic (PLEG): container finished" podID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerID="daf3201c0105e886a4130e243cd4b13a6b3609aef42f1eae7fae271f389043d0" exitCode=0 Sep 11 18:23:07.006609 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:07.006253 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" event={"ID":"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d","Type":"ContainerDied","Data":"daf3201c0105e886a4130e243cd4b13a6b3609aef42f1eae7fae271f389043d0"} Sep 11 18:23:08.138046 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.138020 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:08.186832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.186805 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbrhn\" (UniqueName: \"kubernetes.io/projected/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-kube-api-access-pbrhn\") pod \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " Sep 11 18:23:08.187016 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.186866 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-util\") pod \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " Sep 11 18:23:08.187016 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.186891 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-bundle\") pod \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\" (UID: \"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d\") " Sep 11 18:23:08.187582 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.187550 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-bundle" (OuterVolumeSpecName: "bundle") pod "a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" (UID: "a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:08.188854 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.188831 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-kube-api-access-pbrhn" (OuterVolumeSpecName: "kube-api-access-pbrhn") pod "a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" (UID: "a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d"). InnerVolumeSpecName "kube-api-access-pbrhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:08.195042 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.195014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-util" (OuterVolumeSpecName: "util") pod "a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" (UID: "a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:08.287661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.287584 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbrhn\" (UniqueName: \"kubernetes.io/projected/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-kube-api-access-pbrhn\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:23:08.287661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.287615 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-util\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:23:08.287661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:08.287625 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:23:09.017129 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:09.017096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" event={"ID":"a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d","Type":"ContainerDied","Data":"5ea30c80c6b50c13f1a6dfad5581747c9d4448ddb82e59b5abcdb8dde740db12"} Sep 11 18:23:09.017301 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:09.017134 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmdjb2" Sep 11 18:23:09.017301 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:09.017141 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea30c80c6b50c13f1a6dfad5581747c9d4448ddb82e59b5abcdb8dde740db12" Sep 11 18:23:18.553194 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:18.553153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:18.646899 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:18.646856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:19.041502 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.041468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:19.147589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.147551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:19.387971 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.387885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:19.602037 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.602010 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l"] Sep 11 18:23:19.602393 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.602330 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerName="pull" Sep 11 18:23:19.602393 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.602342 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerName="pull" Sep 11 18:23:19.602393 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.602350 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerName="extract" Sep 11 18:23:19.602393 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.602356 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerName="extract" Sep 11 18:23:19.602393 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.602368 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerName="util" Sep 11 18:23:19.602393 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.602378 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerName="util" Sep 11 18:23:19.602566 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.602432 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43a6cc6-f2f7-4b8d-9ddd-34088ecb0c9d" containerName="extract" Sep 11 18:23:19.606587 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.606570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.609793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.609765 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 18:23:19.610358 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.610341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lgxhw\"" Sep 11 18:23:19.610463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.610342 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 18:23:19.631262 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.631237 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l"] Sep 11 18:23:19.635736 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.635692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:19.676642 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.676578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.676642 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.676619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kmx\" (UniqueName: \"kubernetes.io/projected/5b98abbf-20ba-440a-a863-1338ef0a57bc-kube-api-access-k2kmx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.676800 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.676693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.777684 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.777646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.777883 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.777696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kmx\" (UniqueName: \"kubernetes.io/projected/5b98abbf-20ba-440a-a863-1338ef0a57bc-kube-api-access-k2kmx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.777883 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.777783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.778104 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.778080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.778176 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.778162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.787832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.787798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:19.788369 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.788341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kmx\" (UniqueName: \"kubernetes.io/projected/5b98abbf-20ba-440a-a863-1338ef0a57bc-kube-api-access-k2kmx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.817051 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.817024 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6"] Sep 11 18:23:19.820398 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.820383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.824744 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.824696 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 18:23:19.824987 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.824973 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 18:23:19.825087 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.825069 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 18:23:19.828650 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.828633 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 18:23:19.836520 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.836496 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q6wtr\"" Sep 11 18:23:19.867950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.867922 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6"] Sep 11 18:23:19.878211 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.878185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9dd225af-1468-4cbd-a532-9d0e0fb1a317-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.878315 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.878221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9dd225af-1468-4cbd-a532-9d0e0fb1a317-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.878315 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.878257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8fx\" (UniqueName: \"kubernetes.io/projected/9dd225af-1468-4cbd-a532-9d0e0fb1a317-kube-api-access-fz8fx\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.878424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.878330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9dd225af-1468-4cbd-a532-9d0e0fb1a317-manifests\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.878424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.878373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dd225af-1468-4cbd-a532-9d0e0fb1a317-tmp\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.916173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.916136 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:19.979011 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.978979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8fx\" (UniqueName: \"kubernetes.io/projected/9dd225af-1468-4cbd-a532-9d0e0fb1a317-kube-api-access-fz8fx\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.979147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.979031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9dd225af-1468-4cbd-a532-9d0e0fb1a317-manifests\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.979147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.979059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dd225af-1468-4cbd-a532-9d0e0fb1a317-tmp\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.979147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.979089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9dd225af-1468-4cbd-a532-9d0e0fb1a317-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.979147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.979114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9dd225af-1468-4cbd-a532-9d0e0fb1a317-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.979787 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.979738 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9dd225af-1468-4cbd-a532-9d0e0fb1a317-manifests\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.981946 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.981921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dd225af-1468-4cbd-a532-9d0e0fb1a317-tmp\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.982040 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.982021 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9dd225af-1468-4cbd-a532-9d0e0fb1a317-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.983394 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.983364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9dd225af-1468-4cbd-a532-9d0e0fb1a317-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:19.990752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:19.990721 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8fx\" (UniqueName: \"kubernetes.io/projected/9dd225af-1468-4cbd-a532-9d0e0fb1a317-kube-api-access-fz8fx\") pod \"opendatahub-operator-controller-manager-68b7b6b756-mwrq6\" (UID: \"9dd225af-1468-4cbd-a532-9d0e0fb1a317\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:20.097877 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:20.097839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:20.131337 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:20.131303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:20.207245 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:20.207218 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l"] Sep 11 18:23:20.208254 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:23:20.208226 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b98abbf_20ba_440a_a863_1338ef0a57bc.slice/crio-4e50dea6827e2cd18ab75f61da163fd8562cbace2390090d1878a7a1a8af4495 WatchSource:0}: Error finding container 4e50dea6827e2cd18ab75f61da163fd8562cbace2390090d1878a7a1a8af4495: Status 404 returned error can't find the container with id 4e50dea6827e2cd18ab75f61da163fd8562cbace2390090d1878a7a1a8af4495 Sep 11 18:23:20.225216 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:20.225186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:20.362010 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:20.361974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:20.428978 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:20.428954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6"] Sep 11 18:23:20.430344 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:23:20.430317 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd225af_1468_4cbd_a532_9d0e0fb1a317.slice/crio-8454d68360d8903bf61cfa57fe7c7bb8d946b6e07b7ca51554e3be7840bda9ad WatchSource:0}: Error finding container 8454d68360d8903bf61cfa57fe7c7bb8d946b6e07b7ca51554e3be7840bda9ad: Status 404 returned error can't find the container with id 8454d68360d8903bf61cfa57fe7c7bb8d946b6e07b7ca51554e3be7840bda9ad Sep 11 18:23:20.543487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:20.543451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:20.738452 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:20.738364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:21.078613 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:21.078511 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" event={"ID":"9dd225af-1468-4cbd-a532-9d0e0fb1a317","Type":"ContainerStarted","Data":"8454d68360d8903bf61cfa57fe7c7bb8d946b6e07b7ca51554e3be7840bda9ad"} Sep 11 18:23:21.080595 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:21.080564 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerID="5c30bf33c968df128fed62abebc3173a6c6d7ba8a9b356a74cc32ae712053da1" exitCode=0 Sep 11 18:23:21.080747 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:21.080642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" event={"ID":"5b98abbf-20ba-440a-a863-1338ef0a57bc","Type":"ContainerDied","Data":"5c30bf33c968df128fed62abebc3173a6c6d7ba8a9b356a74cc32ae712053da1"} Sep 11 18:23:21.080747 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:21.080683 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" event={"ID":"5b98abbf-20ba-440a-a863-1338ef0a57bc","Type":"ContainerStarted","Data":"4e50dea6827e2cd18ab75f61da163fd8562cbace2390090d1878a7a1a8af4495"} Sep 11 18:23:22.088308 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:22.088273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" event={"ID":"5b98abbf-20ba-440a-a863-1338ef0a57bc","Type":"ContainerStarted","Data":"ce221c56739d648abec37c01ee6edac36cc05de005924eede432121f8f707761"} Sep 11 18:23:24.100257 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:24.100224 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerID="ce221c56739d648abec37c01ee6edac36cc05de005924eede432121f8f707761" exitCode=0 Sep 11 18:23:24.100660 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:24.100302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" event={"ID":"5b98abbf-20ba-440a-a863-1338ef0a57bc","Type":"ContainerDied","Data":"ce221c56739d648abec37c01ee6edac36cc05de005924eede432121f8f707761"} Sep 11 18:23:24.101983 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:24.101964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" event={"ID":"9dd225af-1468-4cbd-a532-9d0e0fb1a317","Type":"ContainerStarted","Data":"36aae27eb84071296e7abcc446e2b95cbc45dcfd87cca93c92d60da7017d51b1"} Sep 11 18:23:25.111089 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:25.111057 2579 generic.go:356] "Generic (PLEG): container finished" podID="9dd225af-1468-4cbd-a532-9d0e0fb1a317" containerID="36aae27eb84071296e7abcc446e2b95cbc45dcfd87cca93c92d60da7017d51b1" exitCode=0 Sep 11 18:23:25.111519 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:25.111158 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" event={"ID":"9dd225af-1468-4cbd-a532-9d0e0fb1a317","Type":"ContainerDied","Data":"36aae27eb84071296e7abcc446e2b95cbc45dcfd87cca93c92d60da7017d51b1"} Sep 11 18:23:25.113495 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:25.113475 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerID="cbdd808c1935abc25c119076ad2725186961ab0a426855fe582e5df312816dfd" exitCode=0 Sep 11 18:23:25.113588 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:25.113526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" event={"ID":"5b98abbf-20ba-440a-a863-1338ef0a57bc","Type":"ContainerDied","Data":"cbdd808c1935abc25c119076ad2725186961ab0a426855fe582e5df312816dfd"} Sep 11 18:23:26.121799 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.121749 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" event={"ID":"9dd225af-1468-4cbd-a532-9d0e0fb1a317","Type":"ContainerStarted","Data":"b7059a4aaccc77e1d47020444cc9568c67c1f16719f46b024d137393535ecfad"} Sep 11 18:23:26.122290 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.122111 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:26.147411 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.147360 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" podStartSLOduration=2.833061291 podStartE2EDuration="7.147343534s" totalImagesPullingTime="3.304251923s" totalInitContainerRuntime="1.01003032s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:26.145310835 +0000 UTC m=+474.371987175" watchObservedRunningTime="2026-09-11 18:23:26.147343534 +0000 UTC m=+474.374019870" Sep 11 18:23:26.252252 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.252229 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:26.338813 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.338779 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2kmx\" (UniqueName: \"kubernetes.io/projected/5b98abbf-20ba-440a-a863-1338ef0a57bc-kube-api-access-k2kmx\") pod \"5b98abbf-20ba-440a-a863-1338ef0a57bc\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " Sep 11 18:23:26.339058 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.338839 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-bundle\") pod \"5b98abbf-20ba-440a-a863-1338ef0a57bc\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " Sep 11 18:23:26.339058 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.338891 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-util\") pod \"5b98abbf-20ba-440a-a863-1338ef0a57bc\" (UID: \"5b98abbf-20ba-440a-a863-1338ef0a57bc\") " Sep 11 18:23:26.339606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.339575 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-bundle" (OuterVolumeSpecName: "bundle") pod "5b98abbf-20ba-440a-a863-1338ef0a57bc" (UID: "5b98abbf-20ba-440a-a863-1338ef0a57bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:26.341065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.341035 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b98abbf-20ba-440a-a863-1338ef0a57bc-kube-api-access-k2kmx" (OuterVolumeSpecName: "kube-api-access-k2kmx") pod "5b98abbf-20ba-440a-a863-1338ef0a57bc" (UID: "5b98abbf-20ba-440a-a863-1338ef0a57bc"). InnerVolumeSpecName "kube-api-access-k2kmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:26.343147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.343104 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-util" (OuterVolumeSpecName: "util") pod "5b98abbf-20ba-440a-a863-1338ef0a57bc" (UID: "5b98abbf-20ba-440a-a863-1338ef0a57bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:26.440096 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.439990 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:23:26.440096 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.440034 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b98abbf-20ba-440a-a863-1338ef0a57bc-util\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:23:26.440096 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:26.440048 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2kmx\" (UniqueName: \"kubernetes.io/projected/5b98abbf-20ba-440a-a863-1338ef0a57bc-kube-api-access-k2kmx\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:23:27.128523 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:27.128474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" event={"ID":"5b98abbf-20ba-440a-a863-1338ef0a57bc","Type":"ContainerDied","Data":"4e50dea6827e2cd18ab75f61da163fd8562cbace2390090d1878a7a1a8af4495"} Sep 11 18:23:27.128523 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:27.128513 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0nt58l" Sep 11 18:23:27.129000 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:27.128539 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e50dea6827e2cd18ab75f61da163fd8562cbace2390090d1878a7a1a8af4495" Sep 11 18:23:27.234968 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:27.234930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:27.307074 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:27.307027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:32.167967 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:32.167923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:32.274853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:32.274818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:34.800024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:34.799980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:34.898076 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:34.898044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:35.258926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:35.258892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:35.361982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:35.361944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:36.745025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.744995 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-868787cd89-9smjn"] Sep 11 18:23:36.745363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.745289 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerName="util" Sep 11 18:23:36.745363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.745301 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerName="util" Sep 11 18:23:36.745363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.745313 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerName="pull" Sep 11 18:23:36.745363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.745318 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerName="pull" Sep 11 18:23:36.745363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.745327 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerName="extract" Sep 11 18:23:36.745363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.745333 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerName="extract" Sep 11 18:23:36.745559 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.745388 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b98abbf-20ba-440a-a863-1338ef0a57bc" containerName="extract" Sep 11 18:23:36.770463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.770431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:36.771278 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.771254 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868787cd89-9smjn"] Sep 11 18:23:36.820195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.820162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:36.920160 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.920128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:36.932187 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.932151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqfsd\" (UniqueName: \"kubernetes.io/projected/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-kube-api-access-vqfsd\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:36.932279 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.932202 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-oauth-serving-cert\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:36.932279 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.932245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-console-config\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:36.932366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.932282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-console-oauth-config\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:36.932366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.932319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-console-serving-cert\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:36.932366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.932363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-trusted-ca-bundle\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:36.932499 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:36.932383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-service-ca\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.021534 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.021501 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:37.033588 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.033553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-console-oauth-config\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.033858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.033602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-console-serving-cert\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.033858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.033663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-trusted-ca-bundle\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.033858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.033691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-service-ca\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.033858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.033754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqfsd\" (UniqueName: \"kubernetes.io/projected/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-kube-api-access-vqfsd\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.033858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.033805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-oauth-serving-cert\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.033858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.033833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-console-config\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.034719 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.034542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-service-ca\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.034719 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.034568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-oauth-serving-cert\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.034719 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.034642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-console-config\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.034719 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.034658 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-trusted-ca-bundle\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.036237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.036214 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-console-oauth-config\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.036401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.036386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-console-serving-cert\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.043035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.043011 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqfsd\" (UniqueName: \"kubernetes.io/projected/3bf8a114-aaad-4322-b7dd-0bdd82d91a1f-kube-api-access-vqfsd\") pod \"console-868787cd89-9smjn\" (UID: \"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f\") " pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.081063 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.081026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:37.131191 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.131160 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-mwrq6" Sep 11 18:23:37.366480 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.366451 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868787cd89-9smjn"] Sep 11 18:23:37.369061 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:23:37.369033 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf8a114_aaad_4322_b7dd_0bdd82d91a1f.slice/crio-f13386c05fd9551cff4fff7c233b16474df86a6161a13714935b7e5e63207569 WatchSource:0}: Error finding container f13386c05fd9551cff4fff7c233b16474df86a6161a13714935b7e5e63207569: Status 404 returned error can't find the container with id f13386c05fd9551cff4fff7c233b16474df86a6161a13714935b7e5e63207569 Sep 11 18:23:37.534056 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.534002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:37.600866 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:37.600834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:38.180341 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:38.180302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868787cd89-9smjn" event={"ID":"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f","Type":"ContainerStarted","Data":"3f3bb8cf9fed8ff4035a80f5f580b1fca224cffc66189183a08fdcc9172bf089"} Sep 11 18:23:38.180759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:38.180361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868787cd89-9smjn" event={"ID":"3bf8a114-aaad-4322-b7dd-0bdd82d91a1f","Type":"ContainerStarted","Data":"f13386c05fd9551cff4fff7c233b16474df86a6161a13714935b7e5e63207569"} Sep 11 18:23:38.201159 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:38.201107 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-868787cd89-9smjn" podStartSLOduration=2.201090791 podStartE2EDuration="2.201090791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:38.200332785 +0000 UTC m=+486.427009139" watchObservedRunningTime="2026-09-11 18:23:38.201090791 +0000 UTC m=+486.427767131" Sep 11 18:23:39.382167 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.382117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:39.531755 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.531720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:39.847938 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.847904 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74"] Sep 11 18:23:39.850123 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.850101 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:39.853447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.853424 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8c2dz\"" Sep 11 18:23:39.853761 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.853743 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 18:23:39.854465 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.854449 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 18:23:39.854583 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.854456 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 18:23:39.854583 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.854481 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 18:23:39.866691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.866667 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74"] Sep 11 18:23:39.866894 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.866867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:39.962143 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.962110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75441e84-9578-4566-9ed8-1538aee9d55b-tls-certs\") pod \"kube-auth-proxy-5b58fb58cc-bdr74\" (UID: \"75441e84-9578-4566-9ed8-1538aee9d55b\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:39.962325 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.962159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75441e84-9578-4566-9ed8-1538aee9d55b-tmp\") pod \"kube-auth-proxy-5b58fb58cc-bdr74\" (UID: \"75441e84-9578-4566-9ed8-1538aee9d55b\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:39.962325 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:39.962222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5tt\" (UniqueName: \"kubernetes.io/projected/75441e84-9578-4566-9ed8-1538aee9d55b-kube-api-access-5z5tt\") pod \"kube-auth-proxy-5b58fb58cc-bdr74\" (UID: \"75441e84-9578-4566-9ed8-1538aee9d55b\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:40.062921 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:40.062883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75441e84-9578-4566-9ed8-1538aee9d55b-tls-certs\") pod \"kube-auth-proxy-5b58fb58cc-bdr74\" (UID: \"75441e84-9578-4566-9ed8-1538aee9d55b\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:40.063128 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:40.062935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75441e84-9578-4566-9ed8-1538aee9d55b-tmp\") pod \"kube-auth-proxy-5b58fb58cc-bdr74\" (UID: \"75441e84-9578-4566-9ed8-1538aee9d55b\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:40.063128 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:40.062977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5tt\" (UniqueName: \"kubernetes.io/projected/75441e84-9578-4566-9ed8-1538aee9d55b-kube-api-access-5z5tt\") pod \"kube-auth-proxy-5b58fb58cc-bdr74\" (UID: \"75441e84-9578-4566-9ed8-1538aee9d55b\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:40.065255 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:40.065231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75441e84-9578-4566-9ed8-1538aee9d55b-tmp\") pod \"kube-auth-proxy-5b58fb58cc-bdr74\" (UID: \"75441e84-9578-4566-9ed8-1538aee9d55b\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:40.065542 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:40.065524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75441e84-9578-4566-9ed8-1538aee9d55b-tls-certs\") pod \"kube-auth-proxy-5b58fb58cc-bdr74\" (UID: \"75441e84-9578-4566-9ed8-1538aee9d55b\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:40.119151 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:40.119073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5tt\" (UniqueName: \"kubernetes.io/projected/75441e84-9578-4566-9ed8-1538aee9d55b-kube-api-access-5z5tt\") pod \"kube-auth-proxy-5b58fb58cc-bdr74\" (UID: \"75441e84-9578-4566-9ed8-1538aee9d55b\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:40.150285 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:40.150248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:40.160450 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:40.160422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" Sep 11 18:23:40.437244 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:40.437089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74"] Sep 11 18:23:40.439405 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:23:40.439370 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75441e84_9578_4566_9ed8_1538aee9d55b.slice/crio-8fe7c9432b662abf8f4a9a4055fe98eee8d835d635c879a948b6628f2884ae4c WatchSource:0}: Error finding container 8fe7c9432b662abf8f4a9a4055fe98eee8d835d635c879a948b6628f2884ae4c: Status 404 returned error can't find the container with id 8fe7c9432b662abf8f4a9a4055fe98eee8d835d635c879a948b6628f2884ae4c Sep 11 18:23:41.203916 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.203872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" event={"ID":"75441e84-9578-4566-9ed8-1538aee9d55b","Type":"ContainerStarted","Data":"8fe7c9432b662abf8f4a9a4055fe98eee8d835d635c879a948b6628f2884ae4c"} Sep 11 18:23:41.649830 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.649683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:41.652105 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.652052 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc"] Sep 11 18:23:41.655885 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.655865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.658556 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.658533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jdl7c\"" Sep 11 18:23:41.659780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.659568 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 18:23:41.659780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.659624 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 18:23:41.660051 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.659999 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 18:23:41.660162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.660142 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 18:23:41.681531 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.681500 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc"] Sep 11 18:23:41.765147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.765058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:41.781096 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.781053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.781268 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.781122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.781268 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.781181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffst8\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-kube-api-access-ffst8\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.781268 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.781224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.781268 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.781263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.781503 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.781323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.781503 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.781371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233cfe6-cc81-4c3f-9127-21c1434388ab-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.882095 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.882053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.882285 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.882106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233cfe6-cc81-4c3f-9127-21c1434388ab-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.882285 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.882153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.882476 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.882453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.882594 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.882502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffst8\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-kube-api-access-ffst8\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.882594 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.882536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.882594 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.882574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.882813 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.882781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.885919 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.885871 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233cfe6-cc81-4c3f-9127-21c1434388ab-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.886391 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.886164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.886391 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.886354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.886553 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.886481 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.895497 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.895471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.896469 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.896442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffst8\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-kube-api-access-ffst8\") pod \"istiod-openshift-gateway-7b7995d5cd-8w2cc\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:41.969225 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:41.969141 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:42.901004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:42.900968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:43.215304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:43.215196 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" event={"ID":"75441e84-9578-4566-9ed8-1538aee9d55b","Type":"ContainerStarted","Data":"a8615e9359e82ed38262707ed313a17da7608e0cc20dc3f5ce2ef6d2a3fe5b80"} Sep 11 18:23:43.235215 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:43.235168 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-bdr74" podStartSLOduration=1.6600634570000001 podStartE2EDuration="4.235154063s" totalImagesPullingTime="2.575090606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:43.233194803 +0000 UTC m=+491.459871163" watchObservedRunningTime="2026-09-11 18:23:43.235154063 +0000 UTC m=+491.461830403" Sep 11 18:23:43.261918 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:43.261890 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc"] Sep 11 18:23:43.264449 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:23:43.264424 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4233cfe6_cc81_4c3f_9127_21c1434388ab.slice/crio-2d6edb0ef10a124c61d5bee81baa3fdb0a03af6189a5189527ec85c1ffd54f40 WatchSource:0}: Error finding container 2d6edb0ef10a124c61d5bee81baa3fdb0a03af6189a5189527ec85c1ffd54f40: Status 404 returned error can't find the container with id 2d6edb0ef10a124c61d5bee81baa3fdb0a03af6189a5189527ec85c1ffd54f40 Sep 11 18:23:44.230439 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:44.230399 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" event={"ID":"4233cfe6-cc81-4c3f-9127-21c1434388ab","Type":"ContainerStarted","Data":"2d6edb0ef10a124c61d5bee81baa3fdb0a03af6189a5189527ec85c1ffd54f40"} Sep 11 18:23:45.731004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:45.730969 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:23:45.731276 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:45.731035 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:23:46.236475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:46.236435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" event={"ID":"4233cfe6-cc81-4c3f-9127-21c1434388ab","Type":"ContainerStarted","Data":"1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f"} Sep 11 18:23:46.236685 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:46.236612 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:46.238509 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:46.238482 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-8w2cc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:23:46.238639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:46.238537 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" podUID="4233cfe6-cc81-4c3f-9127-21c1434388ab" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:23:46.265930 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:46.265878 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" podStartSLOduration=2.802398638 podStartE2EDuration="5.265862453s" totalImagesPullingTime="2.463463815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:46.262853925 +0000 UTC m=+494.489530278" watchObservedRunningTime="2026-09-11 18:23:46.265862453 +0000 UTC m=+494.492538793" Sep 11 18:23:47.081280 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.081240 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:47.081800 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.081303 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:47.086033 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.086010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:47.242592 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.242554 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-8w2cc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:23:47.242773 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.242624 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" podUID="4233cfe6-cc81-4c3f-9127-21c1434388ab" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:23:47.246035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.246009 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-868787cd89-9smjn" Sep 11 18:23:47.305629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.305584 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b98898d58-2fd8r"] Sep 11 18:23:47.432332 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.432243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:47.530932 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.530897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:47.636687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.636650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:47.734208 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:47.734095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:48.239377 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:48.239327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:48.369872 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:48.369827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:48.467938 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:48.467902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:50.243541 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:50.243512 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:23:50.747143 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:50.747110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:50.849776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:50.849742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:50.946470 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:50.946434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:51.208957 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.208922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:51.255948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.255915 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lfglc"] Sep 11 18:23:51.258915 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.258892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:23:51.263340 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.263321 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 18:23:51.266041 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.266020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4kn\" (UniqueName: \"kubernetes.io/projected/6f118ba9-bb39-4e13-8fd1-2b4ac360259c-kube-api-access-rq4kn\") pod \"model-serving-api-54d89bd5fc-lfglc\" (UID: \"6f118ba9-bb39-4e13-8fd1-2b4ac360259c\") " pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:23:51.266131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.266071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f118ba9-bb39-4e13-8fd1-2b4ac360259c-tls-certs\") pod \"model-serving-api-54d89bd5fc-lfglc\" (UID: \"6f118ba9-bb39-4e13-8fd1-2b4ac360259c\") " pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:23:51.275311 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.275289 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lfglc"] Sep 11 18:23:51.320527 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.320483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:51.366539 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.366500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4kn\" (UniqueName: \"kubernetes.io/projected/6f118ba9-bb39-4e13-8fd1-2b4ac360259c-kube-api-access-rq4kn\") pod \"model-serving-api-54d89bd5fc-lfglc\" (UID: \"6f118ba9-bb39-4e13-8fd1-2b4ac360259c\") " pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:23:51.366763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.366580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f118ba9-bb39-4e13-8fd1-2b4ac360259c-tls-certs\") pod \"model-serving-api-54d89bd5fc-lfglc\" (UID: \"6f118ba9-bb39-4e13-8fd1-2b4ac360259c\") " pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:23:51.369179 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.369145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f118ba9-bb39-4e13-8fd1-2b4ac360259c-tls-certs\") pod \"model-serving-api-54d89bd5fc-lfglc\" (UID: \"6f118ba9-bb39-4e13-8fd1-2b4ac360259c\") " pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:23:51.376111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.376085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4kn\" (UniqueName: \"kubernetes.io/projected/6f118ba9-bb39-4e13-8fd1-2b4ac360259c-kube-api-access-rq4kn\") pod \"model-serving-api-54d89bd5fc-lfglc\" (UID: \"6f118ba9-bb39-4e13-8fd1-2b4ac360259c\") " pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:23:51.420898 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.420757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:51.568254 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.568213 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:23:51.850023 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:51.849950 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lfglc"] Sep 11 18:23:51.853736 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:23:51.853697 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f118ba9_bb39_4e13_8fd1_2b4ac360259c.slice/crio-83b9cc14d2ad3967767605c35ae53ae82af6cfecc51cef53206f7a781f05ff23 WatchSource:0}: Error finding container 83b9cc14d2ad3967767605c35ae53ae82af6cfecc51cef53206f7a781f05ff23: Status 404 returned error can't find the container with id 83b9cc14d2ad3967767605c35ae53ae82af6cfecc51cef53206f7a781f05ff23 Sep 11 18:23:52.265635 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:52.265598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" event={"ID":"6f118ba9-bb39-4e13-8fd1-2b4ac360259c","Type":"ContainerStarted","Data":"83b9cc14d2ad3967767605c35ae53ae82af6cfecc51cef53206f7a781f05ff23"} Sep 11 18:23:53.803648 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:53.803610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:53.869827 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:53.869799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:54.276766 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:54.276733 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" event={"ID":"6f118ba9-bb39-4e13-8fd1-2b4ac360259c","Type":"ContainerStarted","Data":"2c5b4d06124d880205e9c66c9fc4517303e30e120f50c370b65fdde61d9aa14e"} Sep 11 18:23:54.276951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:54.276823 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:23:54.297113 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:23:54.297053 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" podStartSLOduration=1.891597148 podStartE2EDuration="3.297032878s" totalImagesPullingTime="1.40543573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:54.293941399 +0000 UTC m=+502.520617761" watchObservedRunningTime="2026-09-11 18:23:54.297032878 +0000 UTC m=+502.523709218" Sep 11 18:24:01.269576 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.269535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:01.394300 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.394261 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4"] Sep 11 18:24:01.396700 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.396682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:01.399599 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.399472 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 11 18:24:01.400109 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.400089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f9jhk\"" Sep 11 18:24:01.402138 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.402119 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq"] Sep 11 18:24:01.404656 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.404591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:01.407405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.407383 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dlzh6\"" Sep 11 18:24:01.407960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.407939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 18:24:01.415355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.415331 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4"] Sep 11 18:24:01.422518 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.422491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:01.424346 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.424322 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq"] Sep 11 18:24:01.448316 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.448286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83530f82-2858-426f-bcee-886bec79f531-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdcc4\" (UID: \"83530f82-2858-426f-bcee-886bec79f531\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:01.448316 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.448319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6trl\" (UniqueName: \"kubernetes.io/projected/1e545f22-c74a-4d15-a94f-884b2c4e5b24-kube-api-access-f6trl\") pod \"llmisvc-controller-manager-777c6bd7d5-mptwq\" (UID: \"1e545f22-c74a-4d15-a94f-884b2c4e5b24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:01.448483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.448365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgdbb\" (UniqueName: \"kubernetes.io/projected/83530f82-2858-426f-bcee-886bec79f531-kube-api-access-fgdbb\") pod \"kserve-controller-manager-d6f4b65b8-xdcc4\" (UID: \"83530f82-2858-426f-bcee-886bec79f531\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:01.448483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.448402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e545f22-c74a-4d15-a94f-884b2c4e5b24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mptwq\" (UID: \"1e545f22-c74a-4d15-a94f-884b2c4e5b24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:01.549589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.549498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgdbb\" (UniqueName: \"kubernetes.io/projected/83530f82-2858-426f-bcee-886bec79f531-kube-api-access-fgdbb\") pod \"kserve-controller-manager-d6f4b65b8-xdcc4\" (UID: \"83530f82-2858-426f-bcee-886bec79f531\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:01.549589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.549552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e545f22-c74a-4d15-a94f-884b2c4e5b24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mptwq\" (UID: \"1e545f22-c74a-4d15-a94f-884b2c4e5b24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:01.549849 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.549618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83530f82-2858-426f-bcee-886bec79f531-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdcc4\" (UID: \"83530f82-2858-426f-bcee-886bec79f531\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:01.549849 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.549650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6trl\" (UniqueName: \"kubernetes.io/projected/1e545f22-c74a-4d15-a94f-884b2c4e5b24-kube-api-access-f6trl\") pod \"llmisvc-controller-manager-777c6bd7d5-mptwq\" (UID: \"1e545f22-c74a-4d15-a94f-884b2c4e5b24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:01.549849 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:24:01.549694 2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 11 18:24:01.549849 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:24:01.549771 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83530f82-2858-426f-bcee-886bec79f531-cert podName:83530f82-2858-426f-bcee-886bec79f531 nodeName:}" failed. No retries permitted until 2026-09-11 18:24:02.049753325 +0000 UTC m=+510.276429648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/83530f82-2858-426f-bcee-886bec79f531-cert") pod "kserve-controller-manager-d6f4b65b8-xdcc4" (UID: "83530f82-2858-426f-bcee-886bec79f531") : secret "kserve-webhook-server-cert" not found Sep 11 18:24:01.552079 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.552058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e545f22-c74a-4d15-a94f-884b2c4e5b24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mptwq\" (UID: \"1e545f22-c74a-4d15-a94f-884b2c4e5b24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:01.559074 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.559054 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgdbb\" (UniqueName: \"kubernetes.io/projected/83530f82-2858-426f-bcee-886bec79f531-kube-api-access-fgdbb\") pod \"kserve-controller-manager-d6f4b65b8-xdcc4\" (UID: \"83530f82-2858-426f-bcee-886bec79f531\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:01.580369 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.580341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6trl\" (UniqueName: \"kubernetes.io/projected/1e545f22-c74a-4d15-a94f-884b2c4e5b24-kube-api-access-f6trl\") pod \"llmisvc-controller-manager-777c6bd7d5-mptwq\" (UID: \"1e545f22-c74a-4d15-a94f-884b2c4e5b24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:01.665717 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.665666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:01.717808 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.717777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:01.828802 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:01.828697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:02.011519 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:02.011375 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq"] Sep 11 18:24:02.014140 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:24:02.014112 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1e545f22_c74a_4d15_a94f_884b2c4e5b24.slice/crio-9c022c37255f6ce790fe10f6d8a811c5b9e076c9334515aa7c76d591b733bf55 WatchSource:0}: Error finding container 9c022c37255f6ce790fe10f6d8a811c5b9e076c9334515aa7c76d591b733bf55: Status 404 returned error can't find the container with id 9c022c37255f6ce790fe10f6d8a811c5b9e076c9334515aa7c76d591b733bf55 Sep 11 18:24:02.055306 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:02.055277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83530f82-2858-426f-bcee-886bec79f531-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdcc4\" (UID: \"83530f82-2858-426f-bcee-886bec79f531\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:02.057640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:02.057617 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83530f82-2858-426f-bcee-886bec79f531-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdcc4\" (UID: \"83530f82-2858-426f-bcee-886bec79f531\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:02.309749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:02.309699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:02.319769 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:02.319737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" event={"ID":"1e545f22-c74a-4d15-a94f-884b2c4e5b24","Type":"ContainerStarted","Data":"9c022c37255f6ce790fe10f6d8a811c5b9e076c9334515aa7c76d591b733bf55"} Sep 11 18:24:02.606134 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:02.606101 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4"] Sep 11 18:24:02.634159 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:02.634112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:02.686231 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:24:02.686197 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83530f82_2858_426f_bcee_886bec79f531.slice/crio-d495cf5f0f3edae039739425995e1e27c6a6a6b59d49f669b9dbf701a9c6de5c WatchSource:0}: Error finding container d495cf5f0f3edae039739425995e1e27c6a6a6b59d49f669b9dbf701a9c6de5c: Status 404 returned error can't find the container with id d495cf5f0f3edae039739425995e1e27c6a6a6b59d49f669b9dbf701a9c6de5c Sep 11 18:24:02.752586 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:02.752555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:02.860638 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:02.860564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:03.331171 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.331131 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" event={"ID":"83530f82-2858-426f-bcee-886bec79f531","Type":"ContainerStarted","Data":"d495cf5f0f3edae039739425995e1e27c6a6a6b59d49f669b9dbf701a9c6de5c"} Sep 11 18:24:03.509632 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.509596 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc"] Sep 11 18:24:03.509873 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.509840 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" podUID="4233cfe6-cc81-4c3f-9127-21c1434388ab" containerName="discovery" containerID="cri-o://1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f" gracePeriod=30 Sep 11 18:24:03.776031 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.776004 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:24:03.872484 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.872439 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-ca-configmap\") pod \"4233cfe6-cc81-4c3f-9127-21c1434388ab\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " Sep 11 18:24:03.872484 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.872495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-cacerts\") pod \"4233cfe6-cc81-4c3f-9127-21c1434388ab\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " Sep 11 18:24:03.872782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.872527 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-kubeconfig\") pod \"4233cfe6-cc81-4c3f-9127-21c1434388ab\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " Sep 11 18:24:03.872782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.872594 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233cfe6-cc81-4c3f-9127-21c1434388ab-local-certs\") pod \"4233cfe6-cc81-4c3f-9127-21c1434388ab\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " Sep 11 18:24:03.872782 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.872750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-token\") pod \"4233cfe6-cc81-4c3f-9127-21c1434388ab\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " Sep 11 18:24:03.872977 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.872784 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-dns-cert\") pod \"4233cfe6-cc81-4c3f-9127-21c1434388ab\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " Sep 11 18:24:03.872977 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.872825 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffst8\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-kube-api-access-ffst8\") pod \"4233cfe6-cc81-4c3f-9127-21c1434388ab\" (UID: \"4233cfe6-cc81-4c3f-9127-21c1434388ab\") " Sep 11 18:24:03.873085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.872965 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4233cfe6-cc81-4c3f-9127-21c1434388ab" (UID: "4233cfe6-cc81-4c3f-9127-21c1434388ab"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:24:03.873190 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.873168 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-ca-configmap\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:03.875076 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.875049 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4233cfe6-cc81-4c3f-9127-21c1434388ab" (UID: "4233cfe6-cc81-4c3f-9127-21c1434388ab"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:24:03.875190 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.875090 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "4233cfe6-cc81-4c3f-9127-21c1434388ab" (UID: "4233cfe6-cc81-4c3f-9127-21c1434388ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:03.875281 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.875257 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4233cfe6-cc81-4c3f-9127-21c1434388ab-local-certs" (OuterVolumeSpecName: "local-certs") pod "4233cfe6-cc81-4c3f-9127-21c1434388ab" (UID: "4233cfe6-cc81-4c3f-9127-21c1434388ab"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:24:03.875330 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.875257 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-cacerts" (OuterVolumeSpecName: "cacerts") pod "4233cfe6-cc81-4c3f-9127-21c1434388ab" (UID: "4233cfe6-cc81-4c3f-9127-21c1434388ab"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:24:03.875377 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.875340 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4233cfe6-cc81-4c3f-9127-21c1434388ab" (UID: "4233cfe6-cc81-4c3f-9127-21c1434388ab"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:24:03.875418 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.875387 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-kube-api-access-ffst8" (OuterVolumeSpecName: "kube-api-access-ffst8") pod "4233cfe6-cc81-4c3f-9127-21c1434388ab" (UID: "4233cfe6-cc81-4c3f-9127-21c1434388ab"). InnerVolumeSpecName "kube-api-access-ffst8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:03.974026 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.973984 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffst8\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-kube-api-access-ffst8\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:03.974026 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.974023 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-cacerts\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:03.974270 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.974038 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-kubeconfig\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:03.974270 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.974053 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233cfe6-cc81-4c3f-9127-21c1434388ab-local-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:03.974270 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.974066 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:03.974270 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:03.974077 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4233cfe6-cc81-4c3f-9127-21c1434388ab-istio-csr-dns-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:04.339398 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:04.339288 2579 generic.go:356] "Generic (PLEG): container finished" podID="4233cfe6-cc81-4c3f-9127-21c1434388ab" containerID="1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f" exitCode=0 Sep 11 18:24:04.339398 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:04.339338 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" Sep 11 18:24:04.339398 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:04.339392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" event={"ID":"4233cfe6-cc81-4c3f-9127-21c1434388ab","Type":"ContainerDied","Data":"1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f"} Sep 11 18:24:04.340113 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:04.339433 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc" event={"ID":"4233cfe6-cc81-4c3f-9127-21c1434388ab","Type":"ContainerDied","Data":"2d6edb0ef10a124c61d5bee81baa3fdb0a03af6189a5189527ec85c1ffd54f40"} Sep 11 18:24:04.340113 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:04.339457 2579 scope.go:117] "RemoveContainer" containerID="1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f" Sep 11 18:24:04.368348 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:04.368311 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc"] Sep 11 18:24:04.377212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:04.377147 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8w2cc"] Sep 11 18:24:04.575677 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:04.575537 2579 scope.go:117] "RemoveContainer" containerID="1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f" Sep 11 18:24:04.576005 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:24:04.575969 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f\": container with ID starting with 1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f not found: ID does not exist" containerID="1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f" Sep 11 18:24:04.576114 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:04.576008 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f"} err="failed to get container status \"1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f\": rpc error: code = NotFound desc = could not find container \"1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f\": container with ID starting with 1a1c32eb8e198cde41787e04cfb8b936eb5c5b3a62b783f82caaabc61221802f not found: ID does not exist" Sep 11 18:24:05.288537 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:05.288508 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-lfglc" Sep 11 18:24:05.346614 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:05.346573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" event={"ID":"83530f82-2858-426f-bcee-886bec79f531","Type":"ContainerStarted","Data":"4fe286f371746bb2db62b9b9b1e6e992f48fb5029e87c2d2a40418c59aae292b"} Sep 11 18:24:05.347100 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:05.346806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:05.349299 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:05.349274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" event={"ID":"1e545f22-c74a-4d15-a94f-884b2c4e5b24","Type":"ContainerStarted","Data":"103935e7acebc11c2a4296e0e6076455c716a97b58d1652197e002abe656673d"} Sep 11 18:24:05.349427 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:05.349325 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:05.383320 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:05.383258 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" podStartSLOduration=1.886459757 podStartE2EDuration="4.383235001s" totalImagesPullingTime="2.496775244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:05.381544958 +0000 UTC m=+513.608221320" watchObservedRunningTime="2026-09-11 18:24:05.383235001 +0000 UTC m=+513.609911341" Sep 11 18:24:05.426563 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:05.426511 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" podStartSLOduration=1.260542501 podStartE2EDuration="4.42649477s" totalImagesPullingTime="3.165952269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:05.424390649 +0000 UTC m=+513.651067008" watchObservedRunningTime="2026-09-11 18:24:05.42649477 +0000 UTC m=+513.653171167" Sep 11 18:24:06.228144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:06.228107 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4233cfe6-cc81-4c3f-9127-21c1434388ab" path="/var/lib/kubelet/pods/4233cfe6-cc81-4c3f-9127-21c1434388ab/volumes" Sep 11 18:24:12.326335 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.326296 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b98898d58-2fd8r" podUID="45257a96-959d-4e32-a548-ab1577f9f7e1" containerName="console" containerID="cri-o://5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891" gracePeriod=15 Sep 11 18:24:12.578539 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.578483 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:24:12.653871 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.653833 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-console-config\") pod \"45257a96-959d-4e32-a548-ab1577f9f7e1\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " Sep 11 18:24:12.654069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.653885 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-trusted-ca-bundle\") pod \"45257a96-959d-4e32-a548-ab1577f9f7e1\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " Sep 11 18:24:12.654069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.653931 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-service-ca\") pod \"45257a96-959d-4e32-a548-ab1577f9f7e1\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " Sep 11 18:24:12.654069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.653974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-serving-cert\") pod \"45257a96-959d-4e32-a548-ab1577f9f7e1\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " Sep 11 18:24:12.654069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.654030 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-oauth-config\") pod \"45257a96-959d-4e32-a548-ab1577f9f7e1\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " Sep 11 18:24:12.654069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.654068 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-oauth-serving-cert\") pod \"45257a96-959d-4e32-a548-ab1577f9f7e1\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " Sep 11 18:24:12.654321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.654091 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g6pk\" (UniqueName: \"kubernetes.io/projected/45257a96-959d-4e32-a548-ab1577f9f7e1-kube-api-access-4g6pk\") pod \"45257a96-959d-4e32-a548-ab1577f9f7e1\" (UID: \"45257a96-959d-4e32-a548-ab1577f9f7e1\") " Sep 11 18:24:12.654373 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.654328 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "45257a96-959d-4e32-a548-ab1577f9f7e1" (UID: "45257a96-959d-4e32-a548-ab1577f9f7e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:24:12.654373 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.654348 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "45257a96-959d-4e32-a548-ab1577f9f7e1" (UID: "45257a96-959d-4e32-a548-ab1577f9f7e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:24:12.654587 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.654563 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-console-config" (OuterVolumeSpecName: "console-config") pod "45257a96-959d-4e32-a548-ab1577f9f7e1" (UID: "45257a96-959d-4e32-a548-ab1577f9f7e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:24:12.654776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.654748 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "45257a96-959d-4e32-a548-ab1577f9f7e1" (UID: "45257a96-959d-4e32-a548-ab1577f9f7e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:24:12.656417 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.656392 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45257a96-959d-4e32-a548-ab1577f9f7e1-kube-api-access-4g6pk" (OuterVolumeSpecName: "kube-api-access-4g6pk") pod "45257a96-959d-4e32-a548-ab1577f9f7e1" (UID: "45257a96-959d-4e32-a548-ab1577f9f7e1"). InnerVolumeSpecName "kube-api-access-4g6pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:12.656714 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.656679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "45257a96-959d-4e32-a548-ab1577f9f7e1" (UID: "45257a96-959d-4e32-a548-ab1577f9f7e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:24:12.656791 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.656719 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "45257a96-959d-4e32-a548-ab1577f9f7e1" (UID: "45257a96-959d-4e32-a548-ab1577f9f7e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:24:12.755412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.755372 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-trusted-ca-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:12.755412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.755407 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:12.755412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.755417 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:12.755643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.755425 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45257a96-959d-4e32-a548-ab1577f9f7e1-console-oauth-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:12.755643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.755434 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-oauth-serving-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:12.755643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.755442 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g6pk\" (UniqueName: \"kubernetes.io/projected/45257a96-959d-4e32-a548-ab1577f9f7e1-kube-api-access-4g6pk\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:12.755643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:12.755450 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45257a96-959d-4e32-a548-ab1577f9f7e1-console-config\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:13.389286 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:13.389254 2579 generic.go:356] "Generic (PLEG): container finished" podID="45257a96-959d-4e32-a548-ab1577f9f7e1" containerID="5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891" exitCode=0 Sep 11 18:24:13.389687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:13.389349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b98898d58-2fd8r" event={"ID":"45257a96-959d-4e32-a548-ab1577f9f7e1","Type":"ContainerDied","Data":"5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891"} Sep 11 18:24:13.389687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:13.389360 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b98898d58-2fd8r" Sep 11 18:24:13.389687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:13.389382 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b98898d58-2fd8r" event={"ID":"45257a96-959d-4e32-a548-ab1577f9f7e1","Type":"ContainerDied","Data":"594ce418023f09cc81b87a529be718dec51a8d87392f27f3fc0ea20e34701872"} Sep 11 18:24:13.389687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:13.389400 2579 scope.go:117] "RemoveContainer" containerID="5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891" Sep 11 18:24:13.399112 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:13.399095 2579 scope.go:117] "RemoveContainer" containerID="5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891" Sep 11 18:24:13.399422 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:24:13.399389 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891\": container with ID starting with 5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891 not found: ID does not exist" containerID="5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891" Sep 11 18:24:13.399526 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:13.399433 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891"} err="failed to get container status \"5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891\": rpc error: code = NotFound desc = could not find container \"5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891\": container with ID starting with 5723326fa4075ee39b990451dfe6babb4786149905565ce297dbcf562597d891 not found: ID does not exist" Sep 11 18:24:13.413333 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:13.413310 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b98898d58-2fd8r"] Sep 11 18:24:13.418283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:13.418257 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b98898d58-2fd8r"] Sep 11 18:24:13.422913 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:24:13.422890 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45257a96_959d_4e32_a548_ab1577f9f7e1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45257a96_959d_4e32_a548_ab1577f9f7e1.slice/crio-594ce418023f09cc81b87a529be718dec51a8d87392f27f3fc0ea20e34701872\": RecentStats: unable to find data in memory cache]" Sep 11 18:24:14.228239 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:14.228204 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45257a96-959d-4e32-a548-ab1577f9f7e1" path="/var/lib/kubelet/pods/45257a96-959d-4e32-a548-ab1577f9f7e1/volumes" Sep 11 18:24:36.357016 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:36.356983 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mptwq" Sep 11 18:24:36.360004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:36.359982 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdcc4" Sep 11 18:24:39.573669 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:39.573631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:39.674919 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:39.674883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:51.605486 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:51.605441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:51.702352 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:51.702317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:54.641838 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.641804 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf"] Sep 11 18:24:54.642199 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.642157 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4233cfe6-cc81-4c3f-9127-21c1434388ab" containerName="discovery" Sep 11 18:24:54.642199 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.642168 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4233cfe6-cc81-4c3f-9127-21c1434388ab" containerName="discovery" Sep 11 18:24:54.642199 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.642182 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45257a96-959d-4e32-a548-ab1577f9f7e1" containerName="console" Sep 11 18:24:54.642199 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.642187 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45257a96-959d-4e32-a548-ab1577f9f7e1" containerName="console" Sep 11 18:24:54.642333 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.642237 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4233cfe6-cc81-4c3f-9127-21c1434388ab" containerName="discovery" Sep 11 18:24:54.642333 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.642249 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45257a96-959d-4e32-a548-ab1577f9f7e1" containerName="console" Sep 11 18:24:54.648981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.648960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.652074 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.652052 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lgxhw\"" Sep 11 18:24:54.652074 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.652065 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 18:24:54.653041 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.653024 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 18:24:54.657029 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.657007 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf"] Sep 11 18:24:54.708088 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.708046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bq4\" (UniqueName: \"kubernetes.io/projected/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-kube-api-access-76bq4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.708282 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.708097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.708282 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.708211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.743633 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.743599 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj"] Sep 11 18:24:54.747120 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.747104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.757127 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.757102 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj"] Sep 11 18:24:54.809366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.809328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bq4\" (UniqueName: \"kubernetes.io/projected/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-kube-api-access-76bq4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.809366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.809367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.809609 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.809398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.809609 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.809420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.809609 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.809449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.809609 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.809466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vw8t\" (UniqueName: \"kubernetes.io/projected/d935953d-f226-46cc-9f03-9acc56522452-kube-api-access-7vw8t\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.809865 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.809808 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.809865 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.809845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.819278 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.819249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bq4\" (UniqueName: \"kubernetes.io/projected/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-kube-api-access-76bq4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:54.840977 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.840947 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh"] Sep 11 18:24:54.844641 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.844626 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:54.854362 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.854334 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh"] Sep 11 18:24:54.910126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.910028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.910126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.910064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.910126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.910096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vw8t\" (UniqueName: \"kubernetes.io/projected/d935953d-f226-46cc-9f03-9acc56522452-kube-api-access-7vw8t\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.910126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.910125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdl67\" (UniqueName: \"kubernetes.io/projected/238057fa-8959-42a6-938f-1e060ce79196-kube-api-access-vdl67\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:54.910492 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.910150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:54.910492 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.910166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:54.910492 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.910427 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.910492 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.910484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.919316 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.919290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vw8t\" (UniqueName: \"kubernetes.io/projected/d935953d-f226-46cc-9f03-9acc56522452-kube-api-access-7vw8t\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:54.943973 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.943944 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq"] Sep 11 18:24:54.947562 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.947546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:54.958429 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.958406 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq"] Sep 11 18:24:54.958543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:54.958462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:55.011642 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.011581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:55.012120 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.012088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:55.012228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.012210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.012308 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.012240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.012308 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.012285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nwx\" (UniqueName: \"kubernetes.io/projected/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-kube-api-access-98nwx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.012414 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.012341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdl67\" (UniqueName: \"kubernetes.io/projected/238057fa-8959-42a6-938f-1e060ce79196-kube-api-access-vdl67\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:55.012470 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.012019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:55.012828 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.012806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:55.023144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.022898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdl67\" (UniqueName: \"kubernetes.io/projected/238057fa-8959-42a6-938f-1e060ce79196-kube-api-access-vdl67\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:55.056587 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.056549 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:24:55.114457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.113473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nwx\" (UniqueName: \"kubernetes.io/projected/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-kube-api-access-98nwx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.114457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.113603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.114457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.113633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.114457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.114074 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.114457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.114444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.124719 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.124681 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nwx\" (UniqueName: \"kubernetes.io/projected/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-kube-api-access-98nwx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.154525 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.154487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:55.239134 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.239109 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf"] Sep 11 18:24:55.240787 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:24:55.240758 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e325c89_4a5f_419e_9cd6_f6f26f53dcc9.slice/crio-af55757683328f5a5832091bdf5aab5e1566540816c7345ff3e361f3575661ed WatchSource:0}: Error finding container af55757683328f5a5832091bdf5aab5e1566540816c7345ff3e361f3575661ed: Status 404 returned error can't find the container with id af55757683328f5a5832091bdf5aab5e1566540816c7345ff3e361f3575661ed Sep 11 18:24:55.258067 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.258047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:55.337120 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.337090 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj"] Sep 11 18:24:55.339806 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:24:55.339768 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd935953d_f226_46cc_9f03_9acc56522452.slice/crio-3b55381128b5830a6ce9f351a85e16e89e2d903d74fa317bc23385652809dbc6 WatchSource:0}: Error finding container 3b55381128b5830a6ce9f351a85e16e89e2d903d74fa317bc23385652809dbc6: Status 404 returned error can't find the container with id 3b55381128b5830a6ce9f351a85e16e89e2d903d74fa317bc23385652809dbc6 Sep 11 18:24:55.432391 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.432368 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh"] Sep 11 18:24:55.451639 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:24:55.451616 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238057fa_8959_42a6_938f_1e060ce79196.slice/crio-49a5ad3f43dea81ff8a820581ae313f5a27983313f8fffcac7e25d2dc918922a WatchSource:0}: Error finding container 49a5ad3f43dea81ff8a820581ae313f5a27983313f8fffcac7e25d2dc918922a: Status 404 returned error can't find the container with id 49a5ad3f43dea81ff8a820581ae313f5a27983313f8fffcac7e25d2dc918922a Sep 11 18:24:55.534600 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.534566 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq"] Sep 11 18:24:55.589986 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:24:55.589946 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235f0a4e_cdc6_4212_9ce3_d77e0f2d1697.slice/crio-9ad48bd59bdb94d5a0e91dfb645b0fe4c87c251378e83643bc8b9c0dbc7dcd61 WatchSource:0}: Error finding container 9ad48bd59bdb94d5a0e91dfb645b0fe4c87c251378e83643bc8b9c0dbc7dcd61: Status 404 returned error can't find the container with id 9ad48bd59bdb94d5a0e91dfb645b0fe4c87c251378e83643bc8b9c0dbc7dcd61 Sep 11 18:24:55.591715 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.591568 2579 generic.go:356] "Generic (PLEG): container finished" podID="d935953d-f226-46cc-9f03-9acc56522452" containerID="142bfa6f4d5daaccb3abb9370981641ae1c763ad494e44ab90ca605ac11f3379" exitCode=0 Sep 11 18:24:55.591715 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.591606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" event={"ID":"d935953d-f226-46cc-9f03-9acc56522452","Type":"ContainerDied","Data":"142bfa6f4d5daaccb3abb9370981641ae1c763ad494e44ab90ca605ac11f3379"} Sep 11 18:24:55.591715 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.591652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" event={"ID":"d935953d-f226-46cc-9f03-9acc56522452","Type":"ContainerStarted","Data":"3b55381128b5830a6ce9f351a85e16e89e2d903d74fa317bc23385652809dbc6"} Sep 11 18:24:55.593580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.593561 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerID="1c4125f48258c9645052608af079c825f986785ccd0f1f7dd7a0ade4aeddaf4a" exitCode=0 Sep 11 18:24:55.593674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.593650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" event={"ID":"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9","Type":"ContainerDied","Data":"1c4125f48258c9645052608af079c825f986785ccd0f1f7dd7a0ade4aeddaf4a"} Sep 11 18:24:55.593764 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.593697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" event={"ID":"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9","Type":"ContainerStarted","Data":"af55757683328f5a5832091bdf5aab5e1566540816c7345ff3e361f3575661ed"} Sep 11 18:24:55.595421 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.595393 2579 generic.go:356] "Generic (PLEG): container finished" podID="238057fa-8959-42a6-938f-1e060ce79196" containerID="6f945dce92214d6e5e351417e11c4659f70730567b3b10ebd7b662d2b48d756d" exitCode=0 Sep 11 18:24:55.595515 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.595435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" event={"ID":"238057fa-8959-42a6-938f-1e060ce79196","Type":"ContainerDied","Data":"6f945dce92214d6e5e351417e11c4659f70730567b3b10ebd7b662d2b48d756d"} Sep 11 18:24:55.595515 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:55.595471 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" event={"ID":"238057fa-8959-42a6-938f-1e060ce79196","Type":"ContainerStarted","Data":"49a5ad3f43dea81ff8a820581ae313f5a27983313f8fffcac7e25d2dc918922a"} Sep 11 18:24:56.602395 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:56.602361 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerID="dfbce9653b50563ccf426c6615f0c90c79fad1b12489f80a22558047c980546e" exitCode=0 Sep 11 18:24:56.602789 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:56.602443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" event={"ID":"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9","Type":"ContainerDied","Data":"dfbce9653b50563ccf426c6615f0c90c79fad1b12489f80a22558047c980546e"} Sep 11 18:24:56.604383 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:56.604362 2579 generic.go:356] "Generic (PLEG): container finished" podID="238057fa-8959-42a6-938f-1e060ce79196" containerID="ce91f8d795d5f4969b25c5d6b818fa238fdea195b237ed1371decb91a83228a3" exitCode=0 Sep 11 18:24:56.604482 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:56.604444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" event={"ID":"238057fa-8959-42a6-938f-1e060ce79196","Type":"ContainerDied","Data":"ce91f8d795d5f4969b25c5d6b818fa238fdea195b237ed1371decb91a83228a3"} Sep 11 18:24:56.605983 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:56.605961 2579 generic.go:356] "Generic (PLEG): container finished" podID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerID="6d396256ff48084bff10944e94382e576337fa2258c7d9fde20fcc03b3530d99" exitCode=0 Sep 11 18:24:56.606065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:56.606033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" event={"ID":"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697","Type":"ContainerDied","Data":"6d396256ff48084bff10944e94382e576337fa2258c7d9fde20fcc03b3530d99"} Sep 11 18:24:56.606129 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:56.606069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" event={"ID":"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697","Type":"ContainerStarted","Data":"9ad48bd59bdb94d5a0e91dfb645b0fe4c87c251378e83643bc8b9c0dbc7dcd61"} Sep 11 18:24:57.612634 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:57.612607 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerID="00d675be797f084ba2f2c7340e28105b48bf3a4bc652269ff31c552dcfafad9c" exitCode=0 Sep 11 18:24:57.612968 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:57.612685 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" event={"ID":"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9","Type":"ContainerDied","Data":"00d675be797f084ba2f2c7340e28105b48bf3a4bc652269ff31c552dcfafad9c"} Sep 11 18:24:57.614911 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:57.614889 2579 generic.go:356] "Generic (PLEG): container finished" podID="238057fa-8959-42a6-938f-1e060ce79196" containerID="53d6771945d0a2c8194f411df460f16ad57a25cf4655f429f07fca4c0201337b" exitCode=0 Sep 11 18:24:57.615024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:57.614967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" event={"ID":"238057fa-8959-42a6-938f-1e060ce79196","Type":"ContainerDied","Data":"53d6771945d0a2c8194f411df460f16ad57a25cf4655f429f07fca4c0201337b"} Sep 11 18:24:57.616752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:57.616722 2579 generic.go:356] "Generic (PLEG): container finished" podID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerID="859495b359dc5475b72ec2b6c786a642c09b2b9af619f653261e78dee2b9116b" exitCode=0 Sep 11 18:24:57.616839 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:57.616775 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" event={"ID":"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697","Type":"ContainerDied","Data":"859495b359dc5475b72ec2b6c786a642c09b2b9af619f653261e78dee2b9116b"} Sep 11 18:24:58.623556 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.623521 2579 generic.go:356] "Generic (PLEG): container finished" podID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerID="c3bf4610c533ce481f3784ee6c2e54233137aa3b8b6006473355a7a20b58b58f" exitCode=0 Sep 11 18:24:58.624034 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.623603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" event={"ID":"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697","Type":"ContainerDied","Data":"c3bf4610c533ce481f3784ee6c2e54233137aa3b8b6006473355a7a20b58b58f"} Sep 11 18:24:58.758515 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.758493 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:58.788653 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.788626 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:58.853658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.853630 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-util\") pod \"238057fa-8959-42a6-938f-1e060ce79196\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " Sep 11 18:24:58.853811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.853680 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-bundle\") pod \"238057fa-8959-42a6-938f-1e060ce79196\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " Sep 11 18:24:58.853811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.853727 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-bundle\") pod \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " Sep 11 18:24:58.853811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.853759 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76bq4\" (UniqueName: \"kubernetes.io/projected/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-kube-api-access-76bq4\") pod \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " Sep 11 18:24:58.853811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.853775 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdl67\" (UniqueName: \"kubernetes.io/projected/238057fa-8959-42a6-938f-1e060ce79196-kube-api-access-vdl67\") pod \"238057fa-8959-42a6-938f-1e060ce79196\" (UID: \"238057fa-8959-42a6-938f-1e060ce79196\") " Sep 11 18:24:58.854029 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.853840 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-util\") pod \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\" (UID: \"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9\") " Sep 11 18:24:58.854326 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.854291 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-bundle" (OuterVolumeSpecName: "bundle") pod "238057fa-8959-42a6-938f-1e060ce79196" (UID: "238057fa-8959-42a6-938f-1e060ce79196"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:24:58.854560 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.854530 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-bundle" (OuterVolumeSpecName: "bundle") pod "5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" (UID: "5e325c89-4a5f-419e-9cd6-f6f26f53dcc9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:24:58.855985 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.855955 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-kube-api-access-76bq4" (OuterVolumeSpecName: "kube-api-access-76bq4") pod "5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" (UID: "5e325c89-4a5f-419e-9cd6-f6f26f53dcc9"). InnerVolumeSpecName "kube-api-access-76bq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:58.856095 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.856022 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238057fa-8959-42a6-938f-1e060ce79196-kube-api-access-vdl67" (OuterVolumeSpecName: "kube-api-access-vdl67") pod "238057fa-8959-42a6-938f-1e060ce79196" (UID: "238057fa-8959-42a6-938f-1e060ce79196"). InnerVolumeSpecName "kube-api-access-vdl67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:58.859752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.859733 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-util" (OuterVolumeSpecName: "util") pod "238057fa-8959-42a6-938f-1e060ce79196" (UID: "238057fa-8959-42a6-938f-1e060ce79196"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:24:58.862643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.862608 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-util" (OuterVolumeSpecName: "util") pod "5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" (UID: "5e325c89-4a5f-419e-9cd6-f6f26f53dcc9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:24:58.955161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.955077 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-util\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:58.955161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.955108 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/238057fa-8959-42a6-938f-1e060ce79196-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:58.955161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.955118 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:58.955161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.955126 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76bq4\" (UniqueName: \"kubernetes.io/projected/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-kube-api-access-76bq4\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:58.955161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.955137 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdl67\" (UniqueName: \"kubernetes.io/projected/238057fa-8959-42a6-938f-1e060ce79196-kube-api-access-vdl67\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:58.955161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:58.955144 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e325c89-4a5f-419e-9cd6-f6f26f53dcc9-util\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:59.630523 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.630493 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" Sep 11 18:24:59.631138 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.630528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4bnf" event={"ID":"5e325c89-4a5f-419e-9cd6-f6f26f53dcc9","Type":"ContainerDied","Data":"af55757683328f5a5832091bdf5aab5e1566540816c7345ff3e361f3575661ed"} Sep 11 18:24:59.631138 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.630576 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af55757683328f5a5832091bdf5aab5e1566540816c7345ff3e361f3575661ed" Sep 11 18:24:59.632447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.632425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" event={"ID":"238057fa-8959-42a6-938f-1e060ce79196","Type":"ContainerDied","Data":"49a5ad3f43dea81ff8a820581ae313f5a27983313f8fffcac7e25d2dc918922a"} Sep 11 18:24:59.632573 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.632466 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a5ad3f43dea81ff8a820581ae313f5a27983313f8fffcac7e25d2dc918922a" Sep 11 18:24:59.632573 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.632429 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rwmnh" Sep 11 18:24:59.634534 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.634512 2579 generic.go:356] "Generic (PLEG): container finished" podID="d935953d-f226-46cc-9f03-9acc56522452" containerID="9fae93f6d85f8fe6fd86d297187f8c10f22cabc8d09eb79fbabb7d1e618e7aa4" exitCode=0 Sep 11 18:24:59.634642 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.634544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" event={"ID":"d935953d-f226-46cc-9f03-9acc56522452","Type":"ContainerDied","Data":"9fae93f6d85f8fe6fd86d297187f8c10f22cabc8d09eb79fbabb7d1e618e7aa4"} Sep 11 18:24:59.770813 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.770785 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:24:59.862820 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.862787 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98nwx\" (UniqueName: \"kubernetes.io/projected/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-kube-api-access-98nwx\") pod \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " Sep 11 18:24:59.863004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.862833 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-util\") pod \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " Sep 11 18:24:59.863004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.862861 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-bundle\") pod \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\" (UID: \"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697\") " Sep 11 18:24:59.863368 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.863330 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-bundle" (OuterVolumeSpecName: "bundle") pod "235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" (UID: "235f0a4e-cdc6-4212-9ce3-d77e0f2d1697"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:24:59.864993 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.864963 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-kube-api-access-98nwx" (OuterVolumeSpecName: "kube-api-access-98nwx") pod "235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" (UID: "235f0a4e-cdc6-4212-9ce3-d77e0f2d1697"). InnerVolumeSpecName "kube-api-access-98nwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:59.868169 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.868138 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-util" (OuterVolumeSpecName: "util") pod "235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" (UID: "235f0a4e-cdc6-4212-9ce3-d77e0f2d1697"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:24:59.963584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.963502 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98nwx\" (UniqueName: \"kubernetes.io/projected/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-kube-api-access-98nwx\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:59.963584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.963532 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-util\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:24:59.963584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:24:59.963543 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/235f0a4e-cdc6-4212-9ce3-d77e0f2d1697-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:25:00.642181 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:00.642147 2579 generic.go:356] "Generic (PLEG): container finished" podID="d935953d-f226-46cc-9f03-9acc56522452" containerID="a7742af74551a466344e45333a5b0aad1934a4a4b842a809542867e35af9b56a" exitCode=0 Sep 11 18:25:00.642601 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:00.642235 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" event={"ID":"d935953d-f226-46cc-9f03-9acc56522452","Type":"ContainerDied","Data":"a7742af74551a466344e45333a5b0aad1934a4a4b842a809542867e35af9b56a"} Sep 11 18:25:00.644125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:00.644098 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" event={"ID":"235f0a4e-cdc6-4212-9ce3-d77e0f2d1697","Type":"ContainerDied","Data":"9ad48bd59bdb94d5a0e91dfb645b0fe4c87c251378e83643bc8b9c0dbc7dcd61"} Sep 11 18:25:00.644125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:00.644120 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fclnq" Sep 11 18:25:00.644295 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:00.644136 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad48bd59bdb94d5a0e91dfb645b0fe4c87c251378e83643bc8b9c0dbc7dcd61" Sep 11 18:25:01.784489 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.784455 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:25:01.878121 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.878084 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-bundle\") pod \"d935953d-f226-46cc-9f03-9acc56522452\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " Sep 11 18:25:01.878283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.878193 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vw8t\" (UniqueName: \"kubernetes.io/projected/d935953d-f226-46cc-9f03-9acc56522452-kube-api-access-7vw8t\") pod \"d935953d-f226-46cc-9f03-9acc56522452\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " Sep 11 18:25:01.878283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.878213 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-util\") pod \"d935953d-f226-46cc-9f03-9acc56522452\" (UID: \"d935953d-f226-46cc-9f03-9acc56522452\") " Sep 11 18:25:01.878673 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.878649 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-bundle" (OuterVolumeSpecName: "bundle") pod "d935953d-f226-46cc-9f03-9acc56522452" (UID: "d935953d-f226-46cc-9f03-9acc56522452"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:25:01.880309 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.880281 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d935953d-f226-46cc-9f03-9acc56522452-kube-api-access-7vw8t" (OuterVolumeSpecName: "kube-api-access-7vw8t") pod "d935953d-f226-46cc-9f03-9acc56522452" (UID: "d935953d-f226-46cc-9f03-9acc56522452"). InnerVolumeSpecName "kube-api-access-7vw8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:25:01.882768 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.882726 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-util" (OuterVolumeSpecName: "util") pod "d935953d-f226-46cc-9f03-9acc56522452" (UID: "d935953d-f226-46cc-9f03-9acc56522452"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:25:01.979521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.979423 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vw8t\" (UniqueName: \"kubernetes.io/projected/d935953d-f226-46cc-9f03-9acc56522452-kube-api-access-7vw8t\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:25:01.979521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.979459 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-util\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:25:01.979521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:01.979468 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d935953d-f226-46cc-9f03-9acc56522452-bundle\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:25:02.655916 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:02.655884 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" Sep 11 18:25:02.656083 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:02.655892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthlgj" event={"ID":"d935953d-f226-46cc-9f03-9acc56522452","Type":"ContainerDied","Data":"3b55381128b5830a6ce9f351a85e16e89e2d903d74fa317bc23385652809dbc6"} Sep 11 18:25:02.656083 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:02.656011 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b55381128b5830a6ce9f351a85e16e89e2d903d74fa317bc23385652809dbc6" Sep 11 18:25:21.598066 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:21.598030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:22.415229 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:22.415190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:24.925990 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.925951 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-msdcq"] Sep 11 18:25:24.926467 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926449 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerName="pull" Sep 11 18:25:24.926517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926472 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerName="pull" Sep 11 18:25:24.926517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926486 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d935953d-f226-46cc-9f03-9acc56522452" containerName="extract" Sep 11 18:25:24.926517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926495 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d935953d-f226-46cc-9f03-9acc56522452" containerName="extract" Sep 11 18:25:24.926517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926512 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerName="util" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926522 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerName="util" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926543 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="238057fa-8959-42a6-938f-1e060ce79196" containerName="util" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926551 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="238057fa-8959-42a6-938f-1e060ce79196" containerName="util" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926561 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerName="extract" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926569 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerName="extract" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926581 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerName="util" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926588 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerName="util" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926600 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d935953d-f226-46cc-9f03-9acc56522452" containerName="util" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926606 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d935953d-f226-46cc-9f03-9acc56522452" containerName="util" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926612 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="238057fa-8959-42a6-938f-1e060ce79196" containerName="pull" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926618 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="238057fa-8959-42a6-938f-1e060ce79196" containerName="pull" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926623 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerName="extract" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926628 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerName="extract" Sep 11 18:25:24.926640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926642 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerName="pull" Sep 11 18:25:24.927069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926647 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerName="pull" Sep 11 18:25:24.927069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926654 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="238057fa-8959-42a6-938f-1e060ce79196" containerName="extract" Sep 11 18:25:24.927069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926659 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="238057fa-8959-42a6-938f-1e060ce79196" containerName="extract" Sep 11 18:25:24.927069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926665 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d935953d-f226-46cc-9f03-9acc56522452" containerName="pull" Sep 11 18:25:24.927069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926671 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d935953d-f226-46cc-9f03-9acc56522452" containerName="pull" Sep 11 18:25:24.927069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926760 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="238057fa-8959-42a6-938f-1e060ce79196" containerName="extract" Sep 11 18:25:24.927069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926770 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e325c89-4a5f-419e-9cd6-f6f26f53dcc9" containerName="extract" Sep 11 18:25:24.927069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926783 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="235f0a4e-cdc6-4212-9ce3-d77e0f2d1697" containerName="extract" Sep 11 18:25:24.927069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.926789 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d935953d-f226-46cc-9f03-9acc56522452" containerName="extract" Sep 11 18:25:24.929502 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.929485 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" Sep 11 18:25:24.932037 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.932014 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 18:25:24.932183 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.932081 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-b5m54\"" Sep 11 18:25:24.932183 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.932086 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 18:25:24.942713 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:24.942669 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-msdcq"] Sep 11 18:25:25.075759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:25.075724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqn7d\" (UniqueName: \"kubernetes.io/projected/c7eea2f1-60ef-488d-96a0-bb43497e6756-kube-api-access-jqn7d\") pod \"authorino-operator-76b478bd78-msdcq\" (UID: \"c7eea2f1-60ef-488d-96a0-bb43497e6756\") " pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" Sep 11 18:25:25.177039 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:25.176950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqn7d\" (UniqueName: \"kubernetes.io/projected/c7eea2f1-60ef-488d-96a0-bb43497e6756-kube-api-access-jqn7d\") pod \"authorino-operator-76b478bd78-msdcq\" (UID: \"c7eea2f1-60ef-488d-96a0-bb43497e6756\") " pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" Sep 11 18:25:25.186724 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:25.186425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqn7d\" (UniqueName: \"kubernetes.io/projected/c7eea2f1-60ef-488d-96a0-bb43497e6756-kube-api-access-jqn7d\") pod \"authorino-operator-76b478bd78-msdcq\" (UID: \"c7eea2f1-60ef-488d-96a0-bb43497e6756\") " pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" Sep 11 18:25:25.239676 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:25.239634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" Sep 11 18:25:25.461749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:25.461619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:25.548590 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:25.548559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-msdcq"] Sep 11 18:25:25.552319 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:25:25.552293 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eea2f1_60ef_488d_96a0_bb43497e6756.slice/crio-efe17c8de6702a1878a4be5e6d25cb000d0fe691361a6c273706ad0646dd7987 WatchSource:0}: Error finding container efe17c8de6702a1878a4be5e6d25cb000d0fe691361a6c273706ad0646dd7987: Status 404 returned error can't find the container with id efe17c8de6702a1878a4be5e6d25cb000d0fe691361a6c273706ad0646dd7987 Sep 11 18:25:25.601628 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:25.601583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:25.700962 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:25.700916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:25.776897 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:25.776865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" event={"ID":"c7eea2f1-60ef-488d-96a0-bb43497e6756","Type":"ContainerStarted","Data":"efe17c8de6702a1878a4be5e6d25cb000d0fe691361a6c273706ad0646dd7987"} Sep 11 18:25:27.435889 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.435858 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj"] Sep 11 18:25:27.444851 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.444826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:27.451215 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.451188 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fcbj2\"" Sep 11 18:25:27.468980 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.468945 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj"] Sep 11 18:25:27.599191 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.599154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqd8t\" (UniqueName: \"kubernetes.io/projected/1c85cd0b-165a-4004-810e-3bc06a37d88d-kube-api-access-kqd8t\") pod \"limitador-operator-controller-manager-74495c69f8-g6ggj\" (UID: \"1c85cd0b-165a-4004-810e-3bc06a37d88d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:27.601419 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.601386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:27.699776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.699676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqd8t\" (UniqueName: \"kubernetes.io/projected/1c85cd0b-165a-4004-810e-3bc06a37d88d-kube-api-access-kqd8t\") pod \"limitador-operator-controller-manager-74495c69f8-g6ggj\" (UID: \"1c85cd0b-165a-4004-810e-3bc06a37d88d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:27.700521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.700480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:27.710816 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.710790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqd8t\" (UniqueName: \"kubernetes.io/projected/1c85cd0b-165a-4004-810e-3bc06a37d88d-kube-api-access-kqd8t\") pod \"limitador-operator-controller-manager-74495c69f8-g6ggj\" (UID: \"1c85cd0b-165a-4004-810e-3bc06a37d88d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:27.757560 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:27.757521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:28.221604 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:28.221575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:28.401875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:28.401851 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj"] Sep 11 18:25:28.403937 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:25:28.403904 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c85cd0b_165a_4004_810e_3bc06a37d88d.slice/crio-9b2efd50f6920bb1b5ab928fe9ec61db3b8a3bb949bbfb5ef7b46b99938536f3 WatchSource:0}: Error finding container 9b2efd50f6920bb1b5ab928fe9ec61db3b8a3bb949bbfb5ef7b46b99938536f3: Status 404 returned error can't find the container with id 9b2efd50f6920bb1b5ab928fe9ec61db3b8a3bb949bbfb5ef7b46b99938536f3 Sep 11 18:25:28.484866 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:28.484829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:28.575594 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:28.575555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:28.690881 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:28.690778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:28.797354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:28.797298 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" event={"ID":"c7eea2f1-60ef-488d-96a0-bb43497e6756","Type":"ContainerStarted","Data":"d820b906a0331837b6ac1c6674a53d90857611032e95de0c4e5fb558a2721166"} Sep 11 18:25:28.797554 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:28.797471 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" Sep 11 18:25:28.798982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:28.798956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" event={"ID":"1c85cd0b-165a-4004-810e-3bc06a37d88d","Type":"ContainerStarted","Data":"9b2efd50f6920bb1b5ab928fe9ec61db3b8a3bb949bbfb5ef7b46b99938536f3"} Sep 11 18:25:28.836877 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:28.836813 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" podStartSLOduration=2.201187824 podStartE2EDuration="4.836796627s" totalImagesPullingTime="2.635608803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:28.834988392 +0000 UTC m=+597.061664761" watchObservedRunningTime="2026-09-11 18:25:28.836796627 +0000 UTC m=+597.063472966" Sep 11 18:25:30.811343 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:30.811300 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" event={"ID":"1c85cd0b-165a-4004-810e-3bc06a37d88d","Type":"ContainerStarted","Data":"412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493"} Sep 11 18:25:30.811840 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:30.811406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:30.831271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:30.831219 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" podStartSLOduration=2.227855903 podStartE2EDuration="3.831203777s" totalImagesPullingTime="1.603347874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:30.830376723 +0000 UTC m=+599.057053088" watchObservedRunningTime="2026-09-11 18:25:30.831203777 +0000 UTC m=+599.057880117" Sep 11 18:25:31.529307 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.529277 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr"] Sep 11 18:25:31.532960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.532938 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:31.536919 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.536898 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rzr4j\"" Sep 11 18:25:31.550539 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.550512 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr"] Sep 11 18:25:31.635275 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.635236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a28401b-294c-4334-89b8-a7391c13690d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t6pkr\" (UID: \"4a28401b-294c-4334-89b8-a7391c13690d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:31.635448 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.635288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mzs\" (UniqueName: \"kubernetes.io/projected/4a28401b-294c-4334-89b8-a7391c13690d-kube-api-access-s5mzs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t6pkr\" (UID: \"4a28401b-294c-4334-89b8-a7391c13690d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:31.736553 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.736514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a28401b-294c-4334-89b8-a7391c13690d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t6pkr\" (UID: \"4a28401b-294c-4334-89b8-a7391c13690d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:31.736744 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.736563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mzs\" (UniqueName: \"kubernetes.io/projected/4a28401b-294c-4334-89b8-a7391c13690d-kube-api-access-s5mzs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t6pkr\" (UID: \"4a28401b-294c-4334-89b8-a7391c13690d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:31.736903 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.736885 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a28401b-294c-4334-89b8-a7391c13690d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t6pkr\" (UID: \"4a28401b-294c-4334-89b8-a7391c13690d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:31.750185 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.750159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mzs\" (UniqueName: \"kubernetes.io/projected/4a28401b-294c-4334-89b8-a7391c13690d-kube-api-access-s5mzs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t6pkr\" (UID: \"4a28401b-294c-4334-89b8-a7391c13690d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:31.843162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:31.843073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:32.058882 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:32.058840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:32.135826 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:32.135800 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr"] Sep 11 18:25:32.137591 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:25:32.137560 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a28401b_294c_4334_89b8_a7391c13690d.slice/crio-5b376f8a6e82742b96f8a3a039f87ee4f48c47aba0d9c9a14aebdc258b13cef9 WatchSource:0}: Error finding container 5b376f8a6e82742b96f8a3a039f87ee4f48c47aba0d9c9a14aebdc258b13cef9: Status 404 returned error can't find the container with id 5b376f8a6e82742b96f8a3a039f87ee4f48c47aba0d9c9a14aebdc258b13cef9 Sep 11 18:25:32.214774 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:32.214748 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:25:32.214936 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:32.214748 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:25:32.645611 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:32.645563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:32.822159 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:32.822120 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" event={"ID":"4a28401b-294c-4334-89b8-a7391c13690d","Type":"ContainerStarted","Data":"5b376f8a6e82742b96f8a3a039f87ee4f48c47aba0d9c9a14aebdc258b13cef9"} Sep 11 18:25:33.740805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:33.740767 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx"] Sep 11 18:25:33.744752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:33.744721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" Sep 11 18:25:33.747734 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:33.747651 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 11 18:25:33.747940 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:33.747917 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-l2bnn\"" Sep 11 18:25:33.753693 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:33.753666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzf9q\" (UniqueName: \"kubernetes.io/projected/e841beac-389b-4feb-83f0-f365d3e1f162-kube-api-access-mzf9q\") pod \"dns-operator-controller-manager-7b84cd7c9c-bh7nx\" (UID: \"e841beac-389b-4feb-83f0-f365d3e1f162\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" Sep 11 18:25:33.755414 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:33.755394 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx"] Sep 11 18:25:33.854412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:33.854380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzf9q\" (UniqueName: \"kubernetes.io/projected/e841beac-389b-4feb-83f0-f365d3e1f162-kube-api-access-mzf9q\") pod \"dns-operator-controller-manager-7b84cd7c9c-bh7nx\" (UID: \"e841beac-389b-4feb-83f0-f365d3e1f162\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" Sep 11 18:25:33.863940 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:33.863909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzf9q\" (UniqueName: \"kubernetes.io/projected/e841beac-389b-4feb-83f0-f365d3e1f162-kube-api-access-mzf9q\") pod \"dns-operator-controller-manager-7b84cd7c9c-bh7nx\" (UID: \"e841beac-389b-4feb-83f0-f365d3e1f162\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" Sep 11 18:25:34.058127 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:34.058073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" Sep 11 18:25:34.345759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:34.345722 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx"] Sep 11 18:25:34.347394 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:25:34.347359 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode841beac_389b_4feb_83f0_f365d3e1f162.slice/crio-b0e40e871df7d95465cfe43a93517726379689b6cbf94a0641b0cafacd799e69 WatchSource:0}: Error finding container b0e40e871df7d95465cfe43a93517726379689b6cbf94a0641b0cafacd799e69: Status 404 returned error can't find the container with id b0e40e871df7d95465cfe43a93517726379689b6cbf94a0641b0cafacd799e69 Sep 11 18:25:34.835636 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:34.835600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" event={"ID":"e841beac-389b-4feb-83f0-f365d3e1f162","Type":"ContainerStarted","Data":"b0e40e871df7d95465cfe43a93517726379689b6cbf94a0641b0cafacd799e69"} Sep 11 18:25:35.008516 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:35.008470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:35.121196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:35.121100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:35.422001 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:35.421905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:35.523643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:35.523602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:36.851461 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:36.851423 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" event={"ID":"4a28401b-294c-4334-89b8-a7391c13690d","Type":"ContainerStarted","Data":"9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4"} Sep 11 18:25:36.851888 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:36.851559 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:36.876333 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:36.876280 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" podStartSLOduration=1.883055079 podStartE2EDuration="5.876260889s" totalImagesPullingTime="3.99320581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:36.874063302 +0000 UTC m=+605.100739669" watchObservedRunningTime="2026-09-11 18:25:36.876260889 +0000 UTC m=+605.102937230" Sep 11 18:25:37.858861 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:37.858820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" event={"ID":"e841beac-389b-4feb-83f0-f365d3e1f162","Type":"ContainerStarted","Data":"38f170642df0e9bcab50768e08bcd87a144eea4d80cce4b200fb10f298c78112"} Sep 11 18:25:37.859335 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:37.859064 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" Sep 11 18:25:37.879921 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:37.879874 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" podStartSLOduration=2.233592824 podStartE2EDuration="4.879860333s" totalImagesPullingTime="2.646267509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:37.878383912 +0000 UTC m=+606.105060286" watchObservedRunningTime="2026-09-11 18:25:37.879860333 +0000 UTC m=+606.106536673" Sep 11 18:25:38.859940 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.859902 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2"] Sep 11 18:25:38.863957 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.863936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:38.867007 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.866985 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 11 18:25:38.867142 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.867085 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g65q8\"" Sep 11 18:25:38.867221 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.867204 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 11 18:25:38.868958 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.868936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:38.874441 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.874416 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2"] Sep 11 18:25:38.899427 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.899401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1674050-c7ab-4053-a394-fbf98776e0ac-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:38.899565 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.899442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbm72\" (UniqueName: \"kubernetes.io/projected/e1674050-c7ab-4053-a394-fbf98776e0ac-kube-api-access-dbm72\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:38.899565 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:38.899525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1674050-c7ab-4053-a394-fbf98776e0ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:39.000323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.000284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbm72\" (UniqueName: \"kubernetes.io/projected/e1674050-c7ab-4053-a394-fbf98776e0ac-kube-api-access-dbm72\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:39.000518 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.000341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1674050-c7ab-4053-a394-fbf98776e0ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:39.000518 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.000399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1674050-c7ab-4053-a394-fbf98776e0ac-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:39.000518 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:39.000486 2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 11 18:25:39.000700 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:39.000578 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1674050-c7ab-4053-a394-fbf98776e0ac-plugin-serving-cert podName:e1674050-c7ab-4053-a394-fbf98776e0ac nodeName:}" failed. No retries permitted until 2026-09-11 18:25:39.500556632 +0000 UTC m=+607.727232971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/e1674050-c7ab-4053-a394-fbf98776e0ac-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-9xms2" (UID: "e1674050-c7ab-4053-a394-fbf98776e0ac") : secret "plugin-serving-cert" not found Sep 11 18:25:39.001091 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.001069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1674050-c7ab-4053-a394-fbf98776e0ac-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:39.009948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.009927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbm72\" (UniqueName: \"kubernetes.io/projected/e1674050-c7ab-4053-a394-fbf98776e0ac-kube-api-access-dbm72\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:39.076731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.076660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:39.208547 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.208442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:39.309553 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.309506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:39.504869 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.504839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1674050-c7ab-4053-a394-fbf98776e0ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:39.507243 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.507221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1674050-c7ab-4053-a394-fbf98776e0ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9xms2\" (UID: \"e1674050-c7ab-4053-a394-fbf98776e0ac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:39.774498 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.774413 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" Sep 11 18:25:39.807734 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:39.807692 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-msdcq" Sep 11 18:25:40.044677 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:40.044601 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2"] Sep 11 18:25:40.047765 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:25:40.047738 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1674050_c7ab_4053_a394_fbf98776e0ac.slice/crio-916852e6c800cb113dcf0fd31f6d915521a339919160fa933796581a24c10b86 WatchSource:0}: Error finding container 916852e6c800cb113dcf0fd31f6d915521a339919160fa933796581a24c10b86: Status 404 returned error can't find the container with id 916852e6c800cb113dcf0fd31f6d915521a339919160fa933796581a24c10b86 Sep 11 18:25:40.879194 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:40.879162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" event={"ID":"e1674050-c7ab-4053-a394-fbf98776e0ac","Type":"ContainerStarted","Data":"916852e6c800cb113dcf0fd31f6d915521a339919160fa933796581a24c10b86"} Sep 11 18:25:41.818379 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:41.818346 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:43.599936 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:43.599898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:43.696864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:43.696806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:44.908671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:44.908632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" event={"ID":"e1674050-c7ab-4053-a394-fbf98776e0ac","Type":"ContainerStarted","Data":"44c6c91720de1f5536e04bbc105a1a18b84bafe675fba3c19e8c8d27cb5a5125"} Sep 11 18:25:44.926983 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:44.926937 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9xms2" podStartSLOduration=2.26205192 podStartE2EDuration="6.926923079s" totalImagesPullingTime="4.664871159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:44.924870675 +0000 UTC m=+613.151547026" watchObservedRunningTime="2026-09-11 18:25:44.926923079 +0000 UTC m=+613.153599418" Sep 11 18:25:47.861638 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:47.861603 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:48.867758 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:48.867696 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bh7nx" Sep 11 18:25:49.136032 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.135953 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44"] Sep 11 18:25:49.139827 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.139809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:49.168466 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.168434 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44"] Sep 11 18:25:49.191495 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.191462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dm5q\" (UniqueName: \"kubernetes.io/projected/a832b0fb-8cd5-4b74-b544-b5edfeb98164-kube-api-access-9dm5q\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jjj44\" (UID: \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:49.191629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.191516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a832b0fb-8cd5-4b74-b544-b5edfeb98164-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jjj44\" (UID: \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:49.292236 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.292198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dm5q\" (UniqueName: \"kubernetes.io/projected/a832b0fb-8cd5-4b74-b544-b5edfeb98164-kube-api-access-9dm5q\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jjj44\" (UID: \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:49.292438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.292257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a832b0fb-8cd5-4b74-b544-b5edfeb98164-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jjj44\" (UID: \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:49.292626 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.292607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a832b0fb-8cd5-4b74-b544-b5edfeb98164-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jjj44\" (UID: \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:49.296928 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.296903 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44"] Sep 11 18:25:49.297185 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:49.297166 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9dm5q], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" podUID="a832b0fb-8cd5-4b74-b544-b5edfeb98164" Sep 11 18:25:49.315685 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.315652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dm5q\" (UniqueName: \"kubernetes.io/projected/a832b0fb-8cd5-4b74-b544-b5edfeb98164-kube-api-access-9dm5q\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jjj44\" (UID: \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:49.934854 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.934817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:49.942337 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.942314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:49.998015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.997983 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dm5q\" (UniqueName: \"kubernetes.io/projected/a832b0fb-8cd5-4b74-b544-b5edfeb98164-kube-api-access-9dm5q\") pod \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\" (UID: \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\") " Sep 11 18:25:49.998015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.998016 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a832b0fb-8cd5-4b74-b544-b5edfeb98164-extensions-socket-volume\") pod \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\" (UID: \"a832b0fb-8cd5-4b74-b544-b5edfeb98164\") " Sep 11 18:25:49.998318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:49.998293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a832b0fb-8cd5-4b74-b544-b5edfeb98164-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a832b0fb-8cd5-4b74-b544-b5edfeb98164" (UID: "a832b0fb-8cd5-4b74-b544-b5edfeb98164"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:25:50.000094 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.000071 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a832b0fb-8cd5-4b74-b544-b5edfeb98164-kube-api-access-9dm5q" (OuterVolumeSpecName: "kube-api-access-9dm5q") pod "a832b0fb-8cd5-4b74-b544-b5edfeb98164" (UID: "a832b0fb-8cd5-4b74-b544-b5edfeb98164"). InnerVolumeSpecName "kube-api-access-9dm5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:25:50.023912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.023886 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44"] Sep 11 18:25:50.041193 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.041165 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44"] Sep 11 18:25:50.055073 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.055046 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr"] Sep 11 18:25:50.055264 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.055240 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" podUID="4a28401b-294c-4334-89b8-a7391c13690d" containerName="manager" containerID="cri-o://9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4" gracePeriod=2 Sep 11 18:25:50.074528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.074503 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr"] Sep 11 18:25:50.099047 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.099016 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dm5q\" (UniqueName: \"kubernetes.io/projected/a832b0fb-8cd5-4b74-b544-b5edfeb98164-kube-api-access-9dm5q\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:25:50.099181 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.099049 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a832b0fb-8cd5-4b74-b544-b5edfeb98164-extensions-socket-volume\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:25:50.099501 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.099476 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj"] Sep 11 18:25:50.099688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.099668 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" podUID="1c85cd0b-165a-4004-810e-3bc06a37d88d" containerName="manager" containerID="cri-o://412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493" gracePeriod=2 Sep 11 18:25:50.104425 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:50.104388 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t6pkr\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="4a28401b-294c-4334-89b8-a7391c13690d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:50.111424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.111403 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj"] Sep 11 18:25:50.120034 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.120012 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk"] Sep 11 18:25:50.120361 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.120349 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a28401b-294c-4334-89b8-a7391c13690d" containerName="manager" Sep 11 18:25:50.120412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.120362 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a28401b-294c-4334-89b8-a7391c13690d" containerName="manager" Sep 11 18:25:50.120412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.120377 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c85cd0b-165a-4004-810e-3bc06a37d88d" containerName="manager" Sep 11 18:25:50.120412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.120383 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c85cd0b-165a-4004-810e-3bc06a37d88d" containerName="manager" Sep 11 18:25:50.120535 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.120437 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a28401b-294c-4334-89b8-a7391c13690d" containerName="manager" Sep 11 18:25:50.120535 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.120446 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c85cd0b-165a-4004-810e-3bc06a37d88d" containerName="manager" Sep 11 18:25:50.123592 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.123575 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:25:50.126775 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:50.126747 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t6pkr\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="4a28401b-294c-4334-89b8-a7391c13690d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:50.147493 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.147466 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk"] Sep 11 18:25:50.167937 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.167908 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm"] Sep 11 18:25:50.170743 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:50.170687 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g6ggj\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="1c85cd0b-165a-4004-810e-3bc06a37d88d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:50.172063 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.172039 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" Sep 11 18:25:50.191945 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.191895 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm"] Sep 11 18:25:50.199573 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.199549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17aa7750-caa9-4431-9c46-15d608d9eb1f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2zlk\" (UID: \"17aa7750-caa9-4431-9c46-15d608d9eb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:25:50.199669 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.199585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzsm5\" (UniqueName: \"kubernetes.io/projected/17aa7750-caa9-4431-9c46-15d608d9eb1f-kube-api-access-xzsm5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2zlk\" (UID: \"17aa7750-caa9-4431-9c46-15d608d9eb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:25:50.229858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.229824 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a832b0fb-8cd5-4b74-b544-b5edfeb98164" path="/var/lib/kubelet/pods/a832b0fb-8cd5-4b74-b544-b5edfeb98164/volumes" Sep 11 18:25:50.244081 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:50.244046 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t6pkr\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="4a28401b-294c-4334-89b8-a7391c13690d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:50.246421 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:50.246391 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g6ggj\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="1c85cd0b-165a-4004-810e-3bc06a37d88d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:50.292335 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.292288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:50.300554 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.300520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17aa7750-caa9-4431-9c46-15d608d9eb1f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2zlk\" (UID: \"17aa7750-caa9-4431-9c46-15d608d9eb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:25:50.300825 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.300583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j2sm\" (UniqueName: \"kubernetes.io/projected/bb9ec434-bb32-40ca-b3c0-7b3be1ad352b-kube-api-access-7j2sm\") pod \"limitador-operator-controller-manager-74495c69f8-4jqrm\" (UID: \"bb9ec434-bb32-40ca-b3c0-7b3be1ad352b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" Sep 11 18:25:50.300825 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.300615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzsm5\" (UniqueName: \"kubernetes.io/projected/17aa7750-caa9-4431-9c46-15d608d9eb1f-kube-api-access-xzsm5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2zlk\" (UID: \"17aa7750-caa9-4431-9c46-15d608d9eb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:25:50.300963 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.300946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17aa7750-caa9-4431-9c46-15d608d9eb1f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2zlk\" (UID: \"17aa7750-caa9-4431-9c46-15d608d9eb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:25:50.317451 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.317428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzsm5\" (UniqueName: \"kubernetes.io/projected/17aa7750-caa9-4431-9c46-15d608d9eb1f-kube-api-access-xzsm5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g2zlk\" (UID: \"17aa7750-caa9-4431-9c46-15d608d9eb1f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:25:50.342636 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.342610 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:50.344689 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.344674 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:50.386645 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.386611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:50.401976 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.401951 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a28401b-294c-4334-89b8-a7391c13690d-extensions-socket-volume\") pod \"4a28401b-294c-4334-89b8-a7391c13690d\" (UID: \"4a28401b-294c-4334-89b8-a7391c13690d\") " Sep 11 18:25:50.402127 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.401992 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqd8t\" (UniqueName: \"kubernetes.io/projected/1c85cd0b-165a-4004-810e-3bc06a37d88d-kube-api-access-kqd8t\") pod \"1c85cd0b-165a-4004-810e-3bc06a37d88d\" (UID: \"1c85cd0b-165a-4004-810e-3bc06a37d88d\") " Sep 11 18:25:50.402127 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.402056 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5mzs\" (UniqueName: \"kubernetes.io/projected/4a28401b-294c-4334-89b8-a7391c13690d-kube-api-access-s5mzs\") pod \"4a28401b-294c-4334-89b8-a7391c13690d\" (UID: \"4a28401b-294c-4334-89b8-a7391c13690d\") " Sep 11 18:25:50.402246 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.402127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j2sm\" (UniqueName: \"kubernetes.io/projected/bb9ec434-bb32-40ca-b3c0-7b3be1ad352b-kube-api-access-7j2sm\") pod \"limitador-operator-controller-manager-74495c69f8-4jqrm\" (UID: \"bb9ec434-bb32-40ca-b3c0-7b3be1ad352b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" Sep 11 18:25:50.402505 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.402464 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a28401b-294c-4334-89b8-a7391c13690d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4a28401b-294c-4334-89b8-a7391c13690d" (UID: "4a28401b-294c-4334-89b8-a7391c13690d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:25:50.404128 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.404100 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a28401b-294c-4334-89b8-a7391c13690d-kube-api-access-s5mzs" (OuterVolumeSpecName: "kube-api-access-s5mzs") pod "4a28401b-294c-4334-89b8-a7391c13690d" (UID: "4a28401b-294c-4334-89b8-a7391c13690d"). InnerVolumeSpecName "kube-api-access-s5mzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:25:50.404225 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.404164 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c85cd0b-165a-4004-810e-3bc06a37d88d-kube-api-access-kqd8t" (OuterVolumeSpecName: "kube-api-access-kqd8t") pod "1c85cd0b-165a-4004-810e-3bc06a37d88d" (UID: "1c85cd0b-165a-4004-810e-3bc06a37d88d"). InnerVolumeSpecName "kube-api-access-kqd8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:25:50.413210 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.413185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j2sm\" (UniqueName: \"kubernetes.io/projected/bb9ec434-bb32-40ca-b3c0-7b3be1ad352b-kube-api-access-7j2sm\") pod \"limitador-operator-controller-manager-74495c69f8-4jqrm\" (UID: \"bb9ec434-bb32-40ca-b3c0-7b3be1ad352b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" Sep 11 18:25:50.479220 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.479158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:25:50.485212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.485186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" Sep 11 18:25:50.502982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.502950 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a28401b-294c-4334-89b8-a7391c13690d-extensions-socket-volume\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:25:50.502982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.502975 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqd8t\" (UniqueName: \"kubernetes.io/projected/1c85cd0b-165a-4004-810e-3bc06a37d88d-kube-api-access-kqd8t\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:25:50.502982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.502986 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5mzs\" (UniqueName: \"kubernetes.io/projected/4a28401b-294c-4334-89b8-a7391c13690d-kube-api-access-s5mzs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:25:50.759143 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.759119 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk"] Sep 11 18:25:50.761545 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:25:50.761512 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17aa7750_caa9_4431_9c46_15d608d9eb1f.slice/crio-f767d89d9b5c8436d28efa37e65d3d499e08736f6f155d0a1920b0ebb9f2bd67 WatchSource:0}: Error finding container f767d89d9b5c8436d28efa37e65d3d499e08736f6f155d0a1920b0ebb9f2bd67: Status 404 returned error can't find the container with id f767d89d9b5c8436d28efa37e65d3d499e08736f6f155d0a1920b0ebb9f2bd67 Sep 11 18:25:50.762789 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.762694 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm"] Sep 11 18:25:50.764625 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:25:50.764605 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9ec434_bb32_40ca_b3c0_7b3be1ad352b.slice/crio-e954155784e718f0611cff07460235b1e679b937af624ebf360b26c77cd7a5cd WatchSource:0}: Error finding container e954155784e718f0611cff07460235b1e679b937af624ebf360b26c77cd7a5cd: Status 404 returned error can't find the container with id e954155784e718f0611cff07460235b1e679b937af624ebf360b26c77cd7a5cd Sep 11 18:25:50.941382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.941346 2579 generic.go:356] "Generic (PLEG): container finished" podID="1c85cd0b-165a-4004-810e-3bc06a37d88d" containerID="412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493" exitCode=0 Sep 11 18:25:50.941809 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.941390 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g6ggj" Sep 11 18:25:50.941809 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.941443 2579 scope.go:117] "RemoveContainer" containerID="412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493" Sep 11 18:25:50.949892 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.949822 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" event={"ID":"17aa7750-caa9-4431-9c46-15d608d9eb1f","Type":"ContainerStarted","Data":"c5717845bfe597456fd80a560c3f3f858e1dabbeafdd73202ea55de345ff9ae0"} Sep 11 18:25:50.950099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.950077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" event={"ID":"17aa7750-caa9-4431-9c46-15d608d9eb1f","Type":"ContainerStarted","Data":"f767d89d9b5c8436d28efa37e65d3d499e08736f6f155d0a1920b0ebb9f2bd67"} Sep 11 18:25:50.950253 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.950230 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:25:50.952170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.952146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" event={"ID":"bb9ec434-bb32-40ca-b3c0-7b3be1ad352b","Type":"ContainerStarted","Data":"c9df489c4bf0db27257017cefc243d14ad9d5c72db1222333c3256f0515ef568"} Sep 11 18:25:50.952263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.952186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" event={"ID":"bb9ec434-bb32-40ca-b3c0-7b3be1ad352b","Type":"ContainerStarted","Data":"e954155784e718f0611cff07460235b1e679b937af624ebf360b26c77cd7a5cd"} Sep 11 18:25:50.952348 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.952330 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" Sep 11 18:25:50.953842 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.953822 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a28401b-294c-4334-89b8-a7391c13690d" containerID="9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4" exitCode=0 Sep 11 18:25:50.953939 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.953880 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t6pkr" Sep 11 18:25:50.954012 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.953883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jjj44" Sep 11 18:25:50.958622 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.958605 2579 scope.go:117] "RemoveContainer" containerID="412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493" Sep 11 18:25:50.958914 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:50.958897 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493\": container with ID starting with 412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493 not found: ID does not exist" containerID="412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493" Sep 11 18:25:50.958987 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.958924 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493"} err="failed to get container status \"412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493\": rpc error: code = NotFound desc = could not find container \"412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493\": container with ID starting with 412b3f0a7a5230a42885a0a0f81b6b46b053b2b145e1d474c25e1bc891e81493 not found: ID does not exist" Sep 11 18:25:50.958987 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.958944 2579 scope.go:117] "RemoveContainer" containerID="9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4" Sep 11 18:25:50.967697 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.967681 2579 scope.go:117] "RemoveContainer" containerID="9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4" Sep 11 18:25:50.967978 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:25:50.967959 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4\": container with ID starting with 9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4 not found: ID does not exist" containerID="9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4" Sep 11 18:25:50.968044 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.967984 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4"} err="failed to get container status \"9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4\": rpc error: code = NotFound desc = could not find container \"9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4\": container with ID starting with 9beca7530d3a1c3992ccd1f865105675adca4f717101caa513919591db7013d4 not found: ID does not exist" Sep 11 18:25:50.974611 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:50.974570 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" podStartSLOduration=0.974557225 podStartE2EDuration="974.557225ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:50.971476727 +0000 UTC m=+619.198153077" watchObservedRunningTime="2026-09-11 18:25:50.974557225 +0000 UTC m=+619.201233565" Sep 11 18:25:51.001673 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:51.001623 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" podStartSLOduration=1.001606411 podStartE2EDuration="1.001606411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:50.999686729 +0000 UTC m=+619.226363101" watchObservedRunningTime="2026-09-11 18:25:51.001606411 +0000 UTC m=+619.228282763" Sep 11 18:25:52.229475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:52.229434 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c85cd0b-165a-4004-810e-3bc06a37d88d" path="/var/lib/kubelet/pods/1c85cd0b-165a-4004-810e-3bc06a37d88d/volumes" Sep 11 18:25:52.229974 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:52.229949 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a28401b-294c-4334-89b8-a7391c13690d" path="/var/lib/kubelet/pods/4a28401b-294c-4334-89b8-a7391c13690d/volumes" Sep 11 18:25:55.665804 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:55.665694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:55.765803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:55.765771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:56.269809 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:56.269764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:56.370625 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:56.370589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:59.672972 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:59.672929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:59.773235 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:25:59.773197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:01.964304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:01.964277 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4jqrm" Sep 11 18:26:02.969596 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:02.969567 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:26:03.865766 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:03.865732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:03.936769 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:03.936724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:04.077459 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:04.077410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:04.177431 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:04.177332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:04.281902 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:04.281865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:04.390413 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:04.390377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:05.340549 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.340506 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk"] Sep 11 18:26:05.341040 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.340758 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" podUID="17aa7750-caa9-4431-9c46-15d608d9eb1f" containerName="manager" containerID="cri-o://c5717845bfe597456fd80a560c3f3f858e1dabbeafdd73202ea55de345ff9ae0" gracePeriod=10 Sep 11 18:26:05.565020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.564981 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq"] Sep 11 18:26:05.589491 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.589462 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq"] Sep 11 18:26:05.589635 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.589565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:05.738423 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.738384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a74684d-e528-4140-9ec8-bfe7fa37bf71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z4lfq\" (UID: \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:05.738598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.738454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j29z\" (UniqueName: \"kubernetes.io/projected/6a74684d-e528-4140-9ec8-bfe7fa37bf71-kube-api-access-5j29z\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z4lfq\" (UID: \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:05.839473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.839433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a74684d-e528-4140-9ec8-bfe7fa37bf71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z4lfq\" (UID: \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:05.839659 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.839487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j29z\" (UniqueName: \"kubernetes.io/projected/6a74684d-e528-4140-9ec8-bfe7fa37bf71-kube-api-access-5j29z\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z4lfq\" (UID: \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:05.839853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.839834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a74684d-e528-4140-9ec8-bfe7fa37bf71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z4lfq\" (UID: \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:05.851715 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.851682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j29z\" (UniqueName: \"kubernetes.io/projected/6a74684d-e528-4140-9ec8-bfe7fa37bf71-kube-api-access-5j29z\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z4lfq\" (UID: \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:05.908534 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:05.908426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:06.048926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.048845 2579 generic.go:356] "Generic (PLEG): container finished" podID="17aa7750-caa9-4431-9c46-15d608d9eb1f" containerID="c5717845bfe597456fd80a560c3f3f858e1dabbeafdd73202ea55de345ff9ae0" exitCode=0 Sep 11 18:26:06.048926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.048910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" event={"ID":"17aa7750-caa9-4431-9c46-15d608d9eb1f","Type":"ContainerDied","Data":"c5717845bfe597456fd80a560c3f3f858e1dabbeafdd73202ea55de345ff9ae0"} Sep 11 18:26:06.138763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.138733 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:26:06.227855 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:26:06.227826 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a74684d_e528_4140_9ec8_bfe7fa37bf71.slice/crio-c6c0322e4ffd9adba2588203bc69c9b49531a1ffae51015968a9ff36ad88f0c9 WatchSource:0}: Error finding container c6c0322e4ffd9adba2588203bc69c9b49531a1ffae51015968a9ff36ad88f0c9: Status 404 returned error can't find the container with id c6c0322e4ffd9adba2588203bc69c9b49531a1ffae51015968a9ff36ad88f0c9 Sep 11 18:26:06.232508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.232481 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq"] Sep 11 18:26:06.246111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.246085 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17aa7750-caa9-4431-9c46-15d608d9eb1f-extensions-socket-volume\") pod \"17aa7750-caa9-4431-9c46-15d608d9eb1f\" (UID: \"17aa7750-caa9-4431-9c46-15d608d9eb1f\") " Sep 11 18:26:06.246214 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.246141 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzsm5\" (UniqueName: \"kubernetes.io/projected/17aa7750-caa9-4431-9c46-15d608d9eb1f-kube-api-access-xzsm5\") pod \"17aa7750-caa9-4431-9c46-15d608d9eb1f\" (UID: \"17aa7750-caa9-4431-9c46-15d608d9eb1f\") " Sep 11 18:26:06.246500 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.246473 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17aa7750-caa9-4431-9c46-15d608d9eb1f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "17aa7750-caa9-4431-9c46-15d608d9eb1f" (UID: "17aa7750-caa9-4431-9c46-15d608d9eb1f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:26:06.248153 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.248135 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17aa7750-caa9-4431-9c46-15d608d9eb1f-kube-api-access-xzsm5" (OuterVolumeSpecName: "kube-api-access-xzsm5") pod "17aa7750-caa9-4431-9c46-15d608d9eb1f" (UID: "17aa7750-caa9-4431-9c46-15d608d9eb1f"). InnerVolumeSpecName "kube-api-access-xzsm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:26:06.347015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.346990 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzsm5\" (UniqueName: \"kubernetes.io/projected/17aa7750-caa9-4431-9c46-15d608d9eb1f-kube-api-access-xzsm5\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:26:06.347015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:06.347016 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17aa7750-caa9-4431-9c46-15d608d9eb1f-extensions-socket-volume\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:26:07.056661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:07.056626 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" event={"ID":"6a74684d-e528-4140-9ec8-bfe7fa37bf71","Type":"ContainerStarted","Data":"61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2"} Sep 11 18:26:07.056886 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:07.056740 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" event={"ID":"6a74684d-e528-4140-9ec8-bfe7fa37bf71","Type":"ContainerStarted","Data":"c6c0322e4ffd9adba2588203bc69c9b49531a1ffae51015968a9ff36ad88f0c9"} Sep 11 18:26:07.056886 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:07.056771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:07.058006 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:07.057981 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" event={"ID":"17aa7750-caa9-4431-9c46-15d608d9eb1f","Type":"ContainerDied","Data":"f767d89d9b5c8436d28efa37e65d3d499e08736f6f155d0a1920b0ebb9f2bd67"} Sep 11 18:26:07.058120 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:07.058014 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk" Sep 11 18:26:07.058120 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:07.058034 2579 scope.go:117] "RemoveContainer" containerID="c5717845bfe597456fd80a560c3f3f858e1dabbeafdd73202ea55de345ff9ae0" Sep 11 18:26:07.089477 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:07.089425 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" podStartSLOduration=2.089406858 podStartE2EDuration="2.089406858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:07.085981535 +0000 UTC m=+635.312657900" watchObservedRunningTime="2026-09-11 18:26:07.089406858 +0000 UTC m=+635.316083198" Sep 11 18:26:07.106303 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:07.106269 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk"] Sep 11 18:26:07.111876 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:07.111845 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g2zlk"] Sep 11 18:26:08.228983 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:08.228948 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17aa7750-caa9-4431-9c46-15d608d9eb1f" path="/var/lib/kubelet/pods/17aa7750-caa9-4431-9c46-15d608d9eb1f/volumes" Sep 11 18:26:09.277191 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:09.277152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:09.364063 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:09.364020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:10.148525 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:10.148480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:10.246046 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:10.245962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:12.650927 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:12.650886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:12.732515 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:12.732475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:18.067513 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:18.067483 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:26:21.730862 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:21.730821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:22.222659 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:22.222611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:22.337547 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:22.337505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:23.649161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:23.649121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:38.631883 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.631849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:38.724910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.724877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:38.800289 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.800255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:38.883363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.883288 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:26:38.883653 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.883642 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17aa7750-caa9-4431-9c46-15d608d9eb1f" containerName="manager" Sep 11 18:26:38.883730 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.883655 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17aa7750-caa9-4431-9c46-15d608d9eb1f" containerName="manager" Sep 11 18:26:38.883784 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.883750 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17aa7750-caa9-4431-9c46-15d608d9eb1f" containerName="manager" Sep 11 18:26:38.888435 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.888414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:38.892990 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.892962 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 18:26:38.895981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.895955 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:26:38.920312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.920275 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:26:38.921836 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.921796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eab82b8c-cc58-4fb5-8a98-1db16127d5a3-config-file\") pod \"limitador-limitador-84994f7ffc-qd92k\" (UID: \"eab82b8c-cc58-4fb5-8a98-1db16127d5a3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:38.921996 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:38.921892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxwlc\" (UniqueName: \"kubernetes.io/projected/eab82b8c-cc58-4fb5-8a98-1db16127d5a3-kube-api-access-fxwlc\") pod \"limitador-limitador-84994f7ffc-qd92k\" (UID: \"eab82b8c-cc58-4fb5-8a98-1db16127d5a3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:39.022641 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.022606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eab82b8c-cc58-4fb5-8a98-1db16127d5a3-config-file\") pod \"limitador-limitador-84994f7ffc-qd92k\" (UID: \"eab82b8c-cc58-4fb5-8a98-1db16127d5a3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:39.022824 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.022650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxwlc\" (UniqueName: \"kubernetes.io/projected/eab82b8c-cc58-4fb5-8a98-1db16127d5a3-kube-api-access-fxwlc\") pod \"limitador-limitador-84994f7ffc-qd92k\" (UID: \"eab82b8c-cc58-4fb5-8a98-1db16127d5a3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:39.023225 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.023207 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eab82b8c-cc58-4fb5-8a98-1db16127d5a3-config-file\") pod \"limitador-limitador-84994f7ffc-qd92k\" (UID: \"eab82b8c-cc58-4fb5-8a98-1db16127d5a3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:39.031390 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.031363 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxwlc\" (UniqueName: \"kubernetes.io/projected/eab82b8c-cc58-4fb5-8a98-1db16127d5a3-kube-api-access-fxwlc\") pod \"limitador-limitador-84994f7ffc-qd92k\" (UID: \"eab82b8c-cc58-4fb5-8a98-1db16127d5a3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:39.199484 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.199389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:39.467400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.467294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:39.478305 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.478281 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:26:39.480929 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:26:39.480904 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab82b8c_cc58_4fb5_8a98_1db16127d5a3.slice/crio-82041d5fc18dfda5d725c698061670572450dd89d5ab96069c98cd1d0ba42c11 WatchSource:0}: Error finding container 82041d5fc18dfda5d725c698061670572450dd89d5ab96069c98cd1d0ba42c11: Status 404 returned error can't find the container with id 82041d5fc18dfda5d725c698061670572450dd89d5ab96069c98cd1d0ba42c11 Sep 11 18:26:39.541108 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.541067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:39.590672 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.590623 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-x7gbf"] Sep 11 18:26:39.595392 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.595367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-x7gbf" Sep 11 18:26:39.598482 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.598428 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jgm7g\"" Sep 11 18:26:39.601346 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.601322 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-x7gbf"] Sep 11 18:26:39.627080 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.627053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxsz\" (UniqueName: \"kubernetes.io/projected/09d28fb5-c27e-4957-9176-14523bdfc5b9-kube-api-access-chxsz\") pod \"authorino-59c974bd48-x7gbf\" (UID: \"09d28fb5-c27e-4957-9176-14523bdfc5b9\") " pod="kuadrant-system/authorino-59c974bd48-x7gbf" Sep 11 18:26:39.727901 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.727809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxsz\" (UniqueName: \"kubernetes.io/projected/09d28fb5-c27e-4957-9176-14523bdfc5b9-kube-api-access-chxsz\") pod \"authorino-59c974bd48-x7gbf\" (UID: \"09d28fb5-c27e-4957-9176-14523bdfc5b9\") " pod="kuadrant-system/authorino-59c974bd48-x7gbf" Sep 11 18:26:39.737145 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.737122 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxsz\" (UniqueName: \"kubernetes.io/projected/09d28fb5-c27e-4957-9176-14523bdfc5b9-kube-api-access-chxsz\") pod \"authorino-59c974bd48-x7gbf\" (UID: \"09d28fb5-c27e-4957-9176-14523bdfc5b9\") " pod="kuadrant-system/authorino-59c974bd48-x7gbf" Sep 11 18:26:39.906767 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:39.906698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-x7gbf" Sep 11 18:26:40.178653 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:40.178623 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-x7gbf"] Sep 11 18:26:40.180869 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:26:40.180830 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d28fb5_c27e_4957_9176_14523bdfc5b9.slice/crio-548b91cae6e246f1ce31f7cf18a036387a0d7d84561285715ab2780867867e88 WatchSource:0}: Error finding container 548b91cae6e246f1ce31f7cf18a036387a0d7d84561285715ab2780867867e88: Status 404 returned error can't find the container with id 548b91cae6e246f1ce31f7cf18a036387a0d7d84561285715ab2780867867e88 Sep 11 18:26:40.251036 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:40.250998 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" event={"ID":"eab82b8c-cc58-4fb5-8a98-1db16127d5a3","Type":"ContainerStarted","Data":"82041d5fc18dfda5d725c698061670572450dd89d5ab96069c98cd1d0ba42c11"} Sep 11 18:26:40.252150 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:40.252124 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-x7gbf" event={"ID":"09d28fb5-c27e-4957-9176-14523bdfc5b9","Type":"ContainerStarted","Data":"548b91cae6e246f1ce31f7cf18a036387a0d7d84561285715ab2780867867e88"} Sep 11 18:26:41.260910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:41.260873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" event={"ID":"eab82b8c-cc58-4fb5-8a98-1db16127d5a3","Type":"ContainerStarted","Data":"96678c738a35f80d6637eb14fe0e59e3b10592e77f1b6a9f2736905af816715a"} Sep 11 18:26:41.261369 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:41.260988 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:41.280074 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:41.280018 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" podStartSLOduration=2.275701026 podStartE2EDuration="3.279995164s" totalImagesPullingTime="1.004294138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:41.279352832 +0000 UTC m=+669.506029193" watchObservedRunningTime="2026-09-11 18:26:41.279995164 +0000 UTC m=+669.506671505" Sep 11 18:26:42.267264 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:42.267225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-x7gbf" event={"ID":"09d28fb5-c27e-4957-9176-14523bdfc5b9","Type":"ContainerStarted","Data":"20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6"} Sep 11 18:26:42.290195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:42.290142 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-x7gbf" podStartSLOduration=1.648281087 podStartE2EDuration="3.290127202s" totalImagesPullingTime="1.641846115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:42.289524441 +0000 UTC m=+670.516200780" watchObservedRunningTime="2026-09-11 18:26:42.290127202 +0000 UTC m=+670.516803569" Sep 11 18:26:52.268162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:52.268131 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-qd92k" Sep 11 18:26:55.564685 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:55.564628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:55.666795 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:55.666760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:56.028999 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:26:56.028962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:07.923287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:07.923246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:08.601695 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:08.601640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:09.682726 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.682682 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-x7gbf"] Sep 11 18:27:09.683140 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.682880 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-x7gbf" podUID="09d28fb5-c27e-4957-9176-14523bdfc5b9" containerName="authorino" containerID="cri-o://20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6" gracePeriod=30 Sep 11 18:27:09.784136 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.784102 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54df648bd5-kbbfq"] Sep 11 18:27:09.788404 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.788381 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:27:09.791123 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.791101 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 18:27:09.797013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.796989 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54df648bd5-kbbfq"] Sep 11 18:27:09.896349 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.896315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29452\" (UniqueName: \"kubernetes.io/projected/ca529ef3-1e34-4290-b592-34d971a596e4-kube-api-access-29452\") pod \"authorino-54df648bd5-kbbfq\" (UID: \"ca529ef3-1e34-4290-b592-34d971a596e4\") " pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:27:09.896525 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.896363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca529ef3-1e34-4290-b592-34d971a596e4-tls-cert\") pod \"authorino-54df648bd5-kbbfq\" (UID: \"ca529ef3-1e34-4290-b592-34d971a596e4\") " pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:27:09.943439 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.943380 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-x7gbf" Sep 11 18:27:09.997947 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.997913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29452\" (UniqueName: \"kubernetes.io/projected/ca529ef3-1e34-4290-b592-34d971a596e4-kube-api-access-29452\") pod \"authorino-54df648bd5-kbbfq\" (UID: \"ca529ef3-1e34-4290-b592-34d971a596e4\") " pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:27:09.998102 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:09.997955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca529ef3-1e34-4290-b592-34d971a596e4-tls-cert\") pod \"authorino-54df648bd5-kbbfq\" (UID: \"ca529ef3-1e34-4290-b592-34d971a596e4\") " pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:27:10.000411 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.000385 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca529ef3-1e34-4290-b592-34d971a596e4-tls-cert\") pod \"authorino-54df648bd5-kbbfq\" (UID: \"ca529ef3-1e34-4290-b592-34d971a596e4\") " pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:27:10.006326 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.006302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29452\" (UniqueName: \"kubernetes.io/projected/ca529ef3-1e34-4290-b592-34d971a596e4-kube-api-access-29452\") pod \"authorino-54df648bd5-kbbfq\" (UID: \"ca529ef3-1e34-4290-b592-34d971a596e4\") " pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:27:10.099072 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.099037 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:27:10.099239 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.099107 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chxsz\" (UniqueName: \"kubernetes.io/projected/09d28fb5-c27e-4957-9176-14523bdfc5b9-kube-api-access-chxsz\") pod \"09d28fb5-c27e-4957-9176-14523bdfc5b9\" (UID: \"09d28fb5-c27e-4957-9176-14523bdfc5b9\") " Sep 11 18:27:10.101162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.101136 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d28fb5-c27e-4957-9176-14523bdfc5b9-kube-api-access-chxsz" (OuterVolumeSpecName: "kube-api-access-chxsz") pod "09d28fb5-c27e-4957-9176-14523bdfc5b9" (UID: "09d28fb5-c27e-4957-9176-14523bdfc5b9"). InnerVolumeSpecName "kube-api-access-chxsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:27:10.200744 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.200648 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxsz\" (UniqueName: \"kubernetes.io/projected/09d28fb5-c27e-4957-9176-14523bdfc5b9-kube-api-access-chxsz\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:27:10.372717 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.372676 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54df648bd5-kbbfq"] Sep 11 18:27:10.373715 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:27:10.373673 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca529ef3_1e34_4290_b592_34d971a596e4.slice/crio-7c31edc3ee010878bee6e8db76ec79cb85a83a1f2c732fc0d8bfafed4ba7a522 WatchSource:0}: Error finding container 7c31edc3ee010878bee6e8db76ec79cb85a83a1f2c732fc0d8bfafed4ba7a522: Status 404 returned error can't find the container with id 7c31edc3ee010878bee6e8db76ec79cb85a83a1f2c732fc0d8bfafed4ba7a522 Sep 11 18:27:10.375009 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.374993 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:27:10.428930 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.428897 2579 generic.go:356] "Generic (PLEG): container finished" podID="09d28fb5-c27e-4957-9176-14523bdfc5b9" containerID="20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6" exitCode=0 Sep 11 18:27:10.429095 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.428935 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-x7gbf" Sep 11 18:27:10.429095 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.428986 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-x7gbf" event={"ID":"09d28fb5-c27e-4957-9176-14523bdfc5b9","Type":"ContainerDied","Data":"20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6"} Sep 11 18:27:10.429095 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.429025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-x7gbf" event={"ID":"09d28fb5-c27e-4957-9176-14523bdfc5b9","Type":"ContainerDied","Data":"548b91cae6e246f1ce31f7cf18a036387a0d7d84561285715ab2780867867e88"} Sep 11 18:27:10.429095 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.429045 2579 scope.go:117] "RemoveContainer" containerID="20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6" Sep 11 18:27:10.430245 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.430219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54df648bd5-kbbfq" event={"ID":"ca529ef3-1e34-4290-b592-34d971a596e4","Type":"ContainerStarted","Data":"7c31edc3ee010878bee6e8db76ec79cb85a83a1f2c732fc0d8bfafed4ba7a522"} Sep 11 18:27:10.437627 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.437609 2579 scope.go:117] "RemoveContainer" containerID="20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6" Sep 11 18:27:10.437920 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:27:10.437902 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6\": container with ID starting with 20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6 not found: ID does not exist" containerID="20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6" Sep 11 18:27:10.437973 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.437929 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6"} err="failed to get container status \"20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6\": rpc error: code = NotFound desc = could not find container \"20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6\": container with ID starting with 20a982d747dddf33bb3447f051db6f11b1c9de29ef1f369fc06a383440e5eaf6 not found: ID does not exist" Sep 11 18:27:10.446887 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.446865 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-x7gbf"] Sep 11 18:27:10.451344 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:10.451293 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-x7gbf"] Sep 11 18:27:11.437962 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:11.437924 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54df648bd5-kbbfq" event={"ID":"ca529ef3-1e34-4290-b592-34d971a596e4","Type":"ContainerStarted","Data":"449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090"} Sep 11 18:27:11.459579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:11.459530 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54df648bd5-kbbfq" podStartSLOduration=1.925158972 podStartE2EDuration="2.459514768s" totalImagesPullingTime="534.355796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:11.457862711 +0000 UTC m=+699.684539052" watchObservedRunningTime="2026-09-11 18:27:11.459514768 +0000 UTC m=+699.686191133" Sep 11 18:27:12.233255 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:12.233221 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d28fb5-c27e-4957-9176-14523bdfc5b9" path="/var/lib/kubelet/pods/09d28fb5-c27e-4957-9176-14523bdfc5b9/volumes" Sep 11 18:27:14.637499 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:14.637454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:14.718063 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:14.718021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:14.789628 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:14.789590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:15.022768 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:15.022735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:15.185197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:15.185160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:18.752608 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:18.752576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:18.853504 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:18.853473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:50.258646 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:50.258603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:50.351649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:50.351613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:51.676875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:51.676834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:51.806921 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:51.806883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:51.931828 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:51.931749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:52.262690 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:52.262645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:52.429721 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:52.429646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:52.534075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:52.533978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:52.783420 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:52.783384 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:27:53.615351 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:27:53.615309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:04.004538 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:04.004500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:05.261930 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:05.261898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:18.852773 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.852739 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl"] Sep 11 18:28:18.853151 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.853120 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d28fb5-c27e-4957-9176-14523bdfc5b9" containerName="authorino" Sep 11 18:28:18.853151 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.853131 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d28fb5-c27e-4957-9176-14523bdfc5b9" containerName="authorino" Sep 11 18:28:18.853224 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.853201 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d28fb5-c27e-4957-9176-14523bdfc5b9" containerName="authorino" Sep 11 18:28:18.856386 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.856367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.860406 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.860374 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 18:28:18.861317 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.861298 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 18:28:18.861412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.861391 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x8pc2\"" Sep 11 18:28:18.861465 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.861431 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 11 18:28:18.871723 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.870687 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl"] Sep 11 18:28:18.896209 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.896171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74da949f-aa62-4be3-89f4-3996004163bc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.896396 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.896285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.896396 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.896316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.896510 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.896387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skzjb\" (UniqueName: \"kubernetes.io/projected/74da949f-aa62-4be3-89f4-3996004163bc-kube-api-access-skzjb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.896510 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.896442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.896510 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.896471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.997178 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.997141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.997178 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.997183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.997443 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.997207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skzjb\" (UniqueName: \"kubernetes.io/projected/74da949f-aa62-4be3-89f4-3996004163bc-kube-api-access-skzjb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.997443 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.997334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.997443 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.997376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.997443 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.997422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74da949f-aa62-4be3-89f4-3996004163bc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.997667 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.997572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.997755 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.997735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.997800 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.997738 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.999596 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.999568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74da949f-aa62-4be3-89f4-3996004163bc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:18.999780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:18.999759 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74da949f-aa62-4be3-89f4-3996004163bc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:19.005885 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:19.005863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skzjb\" (UniqueName: \"kubernetes.io/projected/74da949f-aa62-4be3-89f4-3996004163bc-kube-api-access-skzjb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl\" (UID: \"74da949f-aa62-4be3-89f4-3996004163bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:19.167994 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:19.167901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:19.445860 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:19.445837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl"] Sep 11 18:28:19.447526 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:28:19.447499 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74da949f_aa62_4be3_89f4_3996004163bc.slice/crio-3ba5db28553fb77829e8ed3b3dd62a1f10f92143456823421e18f0bb3dea66f8 WatchSource:0}: Error finding container 3ba5db28553fb77829e8ed3b3dd62a1f10f92143456823421e18f0bb3dea66f8: Status 404 returned error can't find the container with id 3ba5db28553fb77829e8ed3b3dd62a1f10f92143456823421e18f0bb3dea66f8 Sep 11 18:28:19.810381 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:19.810341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" event={"ID":"74da949f-aa62-4be3-89f4-3996004163bc","Type":"ContainerStarted","Data":"3ba5db28553fb77829e8ed3b3dd62a1f10f92143456823421e18f0bb3dea66f8"} Sep 11 18:28:20.816663 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:20.816627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" event={"ID":"74da949f-aa62-4be3-89f4-3996004163bc","Type":"ContainerStarted","Data":"b4fe69dff129be25831758f5e30a61b12b7efa3ccefec103f6c53a1dde8909cb"} Sep 11 18:28:20.840225 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:20.840176 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" podStartSLOduration=1.853046923 podStartE2EDuration="2.840158636s" totalImagesPullingTime="987.111713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:20.838555775 +0000 UTC m=+769.065232126" watchObservedRunningTime="2026-09-11 18:28:20.840158636 +0000 UTC m=+769.066834999" Sep 11 18:28:27.302056 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:27.301987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:27.663982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:27.663894 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:28:29.168947 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:29.168909 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:29.169328 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:29.168975 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:29.181625 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:29.181598 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:29.878542 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:29.878515 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl" Sep 11 18:28:35.619336 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:35.619300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:42.632007 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:42.631968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:42.733085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:42.733050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:47.658668 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:47.658629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:47.969580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:47.969489 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:28:56.521006 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.520927 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft"] Sep 11 18:28:56.523395 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.523378 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.527508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.527483 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 18:28:56.535638 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.535613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff43b734-61c2-4d36-b1d2-c9343860a4ce-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.535784 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.535655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5js98\" (UniqueName: \"kubernetes.io/projected/ff43b734-61c2-4d36-b1d2-c9343860a4ce-kube-api-access-5js98\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.535784 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.535772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.535942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.535804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.535942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.535837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.535942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.535882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.542801 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.542774 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft"] Sep 11 18:28:56.557155 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.557124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:56.636534 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.636499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.636749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.636541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.636749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.636577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.636749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.636618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.636749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.636663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff43b734-61c2-4d36-b1d2-c9343860a4ce-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.636749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.636697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js98\" (UniqueName: \"kubernetes.io/projected/ff43b734-61c2-4d36-b1d2-c9343860a4ce-kube-api-access-5js98\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.637031 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.637000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.637089 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.637059 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.637089 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.637077 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.638896 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.638874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff43b734-61c2-4d36-b1d2-c9343860a4ce-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.639197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.639179 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff43b734-61c2-4d36-b1d2-c9343860a4ce-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.644826 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.644808 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5js98\" (UniqueName: \"kubernetes.io/projected/ff43b734-61c2-4d36-b1d2-c9343860a4ce-kube-api-access-5js98\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6mqft\" (UID: \"ff43b734-61c2-4d36-b1d2-c9343860a4ce\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:56.834092 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:56.834010 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:28:57.107115 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:57.107090 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft"] Sep 11 18:28:57.109076 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:28:57.109052 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff43b734_61c2_4d36_b1d2_c9343860a4ce.slice/crio-1723fd57b6340e4b43b3a66c1139635a8cb9bda78bdd6bcced33b02131a3efc0 WatchSource:0}: Error finding container 1723fd57b6340e4b43b3a66c1139635a8cb9bda78bdd6bcced33b02131a3efc0: Status 404 returned error can't find the container with id 1723fd57b6340e4b43b3a66c1139635a8cb9bda78bdd6bcced33b02131a3efc0 Sep 11 18:28:57.449508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:57.449428 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:28:58.031198 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:58.031161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" event={"ID":"ff43b734-61c2-4d36-b1d2-c9343860a4ce","Type":"ContainerStarted","Data":"97d146ded5ee97fe1540be939c9fdc2b93c9042e9b0c8961b440f42ce58f7ec1"} Sep 11 18:28:58.031601 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:58.031212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" event={"ID":"ff43b734-61c2-4d36-b1d2-c9343860a4ce","Type":"ContainerStarted","Data":"1723fd57b6340e4b43b3a66c1139635a8cb9bda78bdd6bcced33b02131a3efc0"} Sep 11 18:28:58.051675 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:28:58.051616 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" podStartSLOduration=1.8782522799999999 podStartE2EDuration="2.051593848s" totalImagesPullingTime="173.341568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:58.048852285 +0000 UTC m=+806.275528653" watchObservedRunningTime="2026-09-11 18:28:58.051593848 +0000 UTC m=+806.278270191" Sep 11 18:29:06.834374 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:06.834326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:29:06.834990 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:06.834403 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:29:06.846887 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:06.846860 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:29:07.091279 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:07.091194 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6mqft" Sep 11 18:29:08.059634 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:08.059598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:09.242799 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:09.242760 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:29:36.736699 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:36.736660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:38.154069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:38.154026 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:29:47.131651 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.131612 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq"] Sep 11 18:29:47.135476 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.135454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.137977 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.137952 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 11 18:29:47.155084 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.155057 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq"] Sep 11 18:29:47.187781 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.187737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:47.197750 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.197717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf24m\" (UniqueName: \"kubernetes.io/projected/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-kube-api-access-wf24m\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.197890 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.197758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.197890 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.197780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.197890 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.197800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.197890 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.197835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.198107 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.197962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.299005 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.298964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.299005 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.299009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.299228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.299030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.299228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.299053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.299228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.299104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.299228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.299206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf24m\" (UniqueName: \"kubernetes.io/projected/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-kube-api-access-wf24m\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.299451 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.299393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.299516 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.299451 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.299571 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.299554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.301362 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.301339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.301641 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.301622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.307949 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.307920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf24m\" (UniqueName: \"kubernetes.io/projected/b08a45f6-a5c5-4b06-ad5c-cf19f2d01683-kube-api-access-wf24m\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pjglq\" (UID: \"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.451808 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.451727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:47.725273 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:47.725235 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq"] Sep 11 18:29:47.727174 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:29:47.727142 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb08a45f6_a5c5_4b06_ad5c_cf19f2d01683.slice/crio-256aaf1976f2ecc40ce23b47e32edab6c0e3912f4082c0953efcbe089519576f WatchSource:0}: Error finding container 256aaf1976f2ecc40ce23b47e32edab6c0e3912f4082c0953efcbe089519576f: Status 404 returned error can't find the container with id 256aaf1976f2ecc40ce23b47e32edab6c0e3912f4082c0953efcbe089519576f Sep 11 18:29:48.321522 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:48.321488 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" event={"ID":"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683","Type":"ContainerStarted","Data":"e14003a5f87e1eb0784043a8ee0101d33fd07007c7b5721f1acad9d91888dd20"} Sep 11 18:29:48.321895 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:48.321528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" event={"ID":"b08a45f6-a5c5-4b06-ad5c-cf19f2d01683","Type":"ContainerStarted","Data":"256aaf1976f2ecc40ce23b47e32edab6c0e3912f4082c0953efcbe089519576f"} Sep 11 18:29:48.342057 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:48.342007 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" podStartSLOduration=1.142594216 podStartE2EDuration="1.34199037s" totalImagesPullingTime="199.396154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:48.339398743 +0000 UTC m=+856.566075095" watchObservedRunningTime="2026-09-11 18:29:48.34199037 +0000 UTC m=+856.568666710" Sep 11 18:29:49.143426 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:49.143389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:29:57.452431 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:57.452394 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:57.452836 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:57.452454 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:57.464678 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:57.464652 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:29:58.390030 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:29:58.390003 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pjglq" Sep 11 18:30:08.210213 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:30:08.210172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:14.362621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:30:14.362585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:16.544624 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:30:16.544591 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:30:32.253529 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:30:32.253456 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:30:32.254880 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:30:32.254861 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:31:06.005558 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:06.005510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:32.361459 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.361422 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-748dbd7f6-bz497"] Sep 11 18:31:32.364941 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.364922 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-748dbd7f6-bz497" Sep 11 18:31:32.371811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.371781 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-748dbd7f6-bz497"] Sep 11 18:31:32.453862 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.453824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa-tls-cert\") pod \"authorino-748dbd7f6-bz497\" (UID: \"1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa\") " pod="kuadrant-system/authorino-748dbd7f6-bz497" Sep 11 18:31:32.453862 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.453862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h994\" (UniqueName: \"kubernetes.io/projected/1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa-kube-api-access-4h994\") pod \"authorino-748dbd7f6-bz497\" (UID: \"1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa\") " pod="kuadrant-system/authorino-748dbd7f6-bz497" Sep 11 18:31:32.555166 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.555126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa-tls-cert\") pod \"authorino-748dbd7f6-bz497\" (UID: \"1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa\") " pod="kuadrant-system/authorino-748dbd7f6-bz497" Sep 11 18:31:32.555166 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.555167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h994\" (UniqueName: \"kubernetes.io/projected/1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa-kube-api-access-4h994\") pod \"authorino-748dbd7f6-bz497\" (UID: \"1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa\") " pod="kuadrant-system/authorino-748dbd7f6-bz497" Sep 11 18:31:32.557665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.557640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa-tls-cert\") pod \"authorino-748dbd7f6-bz497\" (UID: \"1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa\") " pod="kuadrant-system/authorino-748dbd7f6-bz497" Sep 11 18:31:32.564693 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.564668 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h994\" (UniqueName: \"kubernetes.io/projected/1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa-kube-api-access-4h994\") pod \"authorino-748dbd7f6-bz497\" (UID: \"1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa\") " pod="kuadrant-system/authorino-748dbd7f6-bz497" Sep 11 18:31:32.675233 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.675134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-748dbd7f6-bz497" Sep 11 18:31:32.949225 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:32.949196 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-748dbd7f6-bz497"] Sep 11 18:31:32.951510 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:31:32.951482 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb8f7e7_5cc9_4f6e_b50f_55c03c9b47fa.slice/crio-d0b2b51f0128d2022e3e11e9230d3d53a8ee7d60093473d0e923eb13bcf013cc WatchSource:0}: Error finding container d0b2b51f0128d2022e3e11e9230d3d53a8ee7d60093473d0e923eb13bcf013cc: Status 404 returned error can't find the container with id d0b2b51f0128d2022e3e11e9230d3d53a8ee7d60093473d0e923eb13bcf013cc Sep 11 18:31:33.915989 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:33.915946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-748dbd7f6-bz497" event={"ID":"1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa","Type":"ContainerStarted","Data":"d0b2b51f0128d2022e3e11e9230d3d53a8ee7d60093473d0e923eb13bcf013cc"} Sep 11 18:31:40.959273 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:40.959183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-748dbd7f6-bz497" event={"ID":"1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa","Type":"ContainerStarted","Data":"43fd4b953a0f6bbbedbbb1fa8466ef771639fa145d934d18590d1bad50e717f4"} Sep 11 18:31:40.977274 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:40.977207 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-748dbd7f6-bz497" podStartSLOduration=1.354163605 podStartE2EDuration="8.977188855s" totalImagesPullingTime="7.62302525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:40.975490486 +0000 UTC m=+969.202166827" watchObservedRunningTime="2026-09-11 18:31:40.977188855 +0000 UTC m=+969.203865196" Sep 11 18:31:41.007950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.007915 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54df648bd5-kbbfq"] Sep 11 18:31:41.008196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.008164 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-54df648bd5-kbbfq" podUID="ca529ef3-1e34-4290-b592-34d971a596e4" containerName="authorino" containerID="cri-o://449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090" gracePeriod=30 Sep 11 18:31:41.273632 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.273583 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:31:41.334880 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.334844 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca529ef3-1e34-4290-b592-34d971a596e4-tls-cert\") pod \"ca529ef3-1e34-4290-b592-34d971a596e4\" (UID: \"ca529ef3-1e34-4290-b592-34d971a596e4\") " Sep 11 18:31:41.335074 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.334986 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29452\" (UniqueName: \"kubernetes.io/projected/ca529ef3-1e34-4290-b592-34d971a596e4-kube-api-access-29452\") pod \"ca529ef3-1e34-4290-b592-34d971a596e4\" (UID: \"ca529ef3-1e34-4290-b592-34d971a596e4\") " Sep 11 18:31:41.337043 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.337007 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca529ef3-1e34-4290-b592-34d971a596e4-kube-api-access-29452" (OuterVolumeSpecName: "kube-api-access-29452") pod "ca529ef3-1e34-4290-b592-34d971a596e4" (UID: "ca529ef3-1e34-4290-b592-34d971a596e4"). InnerVolumeSpecName "kube-api-access-29452". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:31:41.345644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.345599 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca529ef3-1e34-4290-b592-34d971a596e4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ca529ef3-1e34-4290-b592-34d971a596e4" (UID: "ca529ef3-1e34-4290-b592-34d971a596e4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:31:41.436580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.436534 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29452\" (UniqueName: \"kubernetes.io/projected/ca529ef3-1e34-4290-b592-34d971a596e4-kube-api-access-29452\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:31:41.436580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.436567 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca529ef3-1e34-4290-b592-34d971a596e4-tls-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:31:41.501409 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.501376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:41.598851 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.598810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:41.966669 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.966586 2579 generic.go:356] "Generic (PLEG): container finished" podID="ca529ef3-1e34-4290-b592-34d971a596e4" containerID="449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090" exitCode=0 Sep 11 18:31:41.966669 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.966628 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df648bd5-kbbfq" Sep 11 18:31:41.967177 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.966668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54df648bd5-kbbfq" event={"ID":"ca529ef3-1e34-4290-b592-34d971a596e4","Type":"ContainerDied","Data":"449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090"} Sep 11 18:31:41.967177 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.966719 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54df648bd5-kbbfq" event={"ID":"ca529ef3-1e34-4290-b592-34d971a596e4","Type":"ContainerDied","Data":"7c31edc3ee010878bee6e8db76ec79cb85a83a1f2c732fc0d8bfafed4ba7a522"} Sep 11 18:31:41.967177 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.966740 2579 scope.go:117] "RemoveContainer" containerID="449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090" Sep 11 18:31:41.977145 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.977122 2579 scope.go:117] "RemoveContainer" containerID="449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090" Sep 11 18:31:41.977514 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:31:41.977472 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090\": container with ID starting with 449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090 not found: ID does not exist" containerID="449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090" Sep 11 18:31:41.977598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.977509 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090"} err="failed to get container status \"449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090\": rpc error: code = NotFound desc = could not find container \"449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090\": container with ID starting with 449be502b9ecec9ac2b4d47a6e088cfa4ea76672e95bdbb9b27dafc528d5a090 not found: ID does not exist" Sep 11 18:31:41.990445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.990409 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54df648bd5-kbbfq"] Sep 11 18:31:41.993828 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:41.993801 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54df648bd5-kbbfq"] Sep 11 18:31:42.229452 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:42.229370 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca529ef3-1e34-4290-b592-34d971a596e4" path="/var/lib/kubelet/pods/ca529ef3-1e34-4290-b592-34d971a596e4/volumes" Sep 11 18:31:42.576856 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:42.576824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:42.684560 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:42.684521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:58.100401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:58.100317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:58.200409 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:58.200369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:58.339861 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:58.339826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:58.633831 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:58.633797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:58.727585 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:58.727544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:58.805532 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:58.805500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:58.874539 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:58.874499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:58.975484 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:58.975405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:59.185485 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:59.185449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:59.524521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:59.524486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:59.599744 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:31:59.599685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:00.023637 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.023604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:00.099960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.099920 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7828-83e63d"] Sep 11 18:32:00.100381 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.100364 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca529ef3-1e34-4290-b592-34d971a596e4" containerName="authorino" Sep 11 18:32:00.100475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.100385 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca529ef3-1e34-4290-b592-34d971a596e4" containerName="authorino" Sep 11 18:32:00.100540 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.100521 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca529ef3-1e34-4290-b592-34d971a596e4" containerName="authorino" Sep 11 18:32:00.103617 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.103597 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-83e63d" Sep 11 18:32:00.106143 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.106117 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vvh9z\"" Sep 11 18:32:00.108680 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.108658 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-83e63d"] Sep 11 18:32:00.207538 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.207506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pq6\" (UniqueName: \"kubernetes.io/projected/f8419743-e2a3-4221-8aac-018e29ca1ca2-kube-api-access-g5pq6\") pod \"test-curl-7828-83e63d\" (UID: \"f8419743-e2a3-4221-8aac-018e29ca1ca2\") " pod="opendatahub/test-curl-7828-83e63d" Sep 11 18:32:00.308765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.308669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pq6\" (UniqueName: \"kubernetes.io/projected/f8419743-e2a3-4221-8aac-018e29ca1ca2-kube-api-access-g5pq6\") pod \"test-curl-7828-83e63d\" (UID: \"f8419743-e2a3-4221-8aac-018e29ca1ca2\") " pod="opendatahub/test-curl-7828-83e63d" Sep 11 18:32:00.316940 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.316903 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pq6\" (UniqueName: \"kubernetes.io/projected/f8419743-e2a3-4221-8aac-018e29ca1ca2-kube-api-access-g5pq6\") pod \"test-curl-7828-83e63d\" (UID: \"f8419743-e2a3-4221-8aac-018e29ca1ca2\") " pod="opendatahub/test-curl-7828-83e63d" Sep 11 18:32:00.413720 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.413680 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-83e63d" Sep 11 18:32:00.685145 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:00.685111 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-83e63d"] Sep 11 18:32:00.686311 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:32:00.686282 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8419743_e2a3_4221_8aac_018e29ca1ca2.slice/crio-7a09f4a0a56c57030a19a152d8adf3fb226bef2978c9641dab65eec711a54398 WatchSource:0}: Error finding container 7a09f4a0a56c57030a19a152d8adf3fb226bef2978c9641dab65eec711a54398: Status 404 returned error can't find the container with id 7a09f4a0a56c57030a19a152d8adf3fb226bef2978c9641dab65eec711a54398 Sep 11 18:32:01.083193 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:01.083153 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-83e63d" event={"ID":"f8419743-e2a3-4221-8aac-018e29ca1ca2","Type":"ContainerStarted","Data":"7a09f4a0a56c57030a19a152d8adf3fb226bef2978c9641dab65eec711a54398"} Sep 11 18:32:01.552687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:01.552653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:01.719613 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:01.719527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:01.847191 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:01.847054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:01.951359 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:01.951313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:02.258372 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:02.258339 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:32:02.718341 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:02.718161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:02.818395 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:02.818353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:02.921354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:02.921315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:03.100613 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.100575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-83e63d" event={"ID":"f8419743-e2a3-4221-8aac-018e29ca1ca2","Type":"ContainerStarted","Data":"d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6"} Sep 11 18:32:03.120739 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.120675 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7828-83e63d" podStartSLOduration=0.797478776 podStartE2EDuration="3.120656492s" totalImagesPullingTime="2.323177716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:03.119491796 +0000 UTC m=+991.346168136" watchObservedRunningTime="2026-09-11 18:32:03.120656492 +0000 UTC m=+991.347332832" Sep 11 18:32:03.414297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.414209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:03.515859 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.515821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:03.706023 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.705943 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7828-83e63d"] Sep 11 18:32:03.710459 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.710429 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7828-83e63d"] Sep 11 18:32:03.789237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.789201 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz"] Sep 11 18:32:03.790499 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.790362 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8419743-e2a3-4221-8aac-018e29ca1ca2" containerName="test-curl-7828-83e63d" Sep 11 18:32:03.790499 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.790386 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8419743-e2a3-4221-8aac-018e29ca1ca2" containerName="test-curl-7828-83e63d" Sep 11 18:32:03.790673 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.790545 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8419743-e2a3-4221-8aac-018e29ca1ca2" containerName="test-curl-7828-83e63d" Sep 11 18:32:03.794221 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.794200 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.797582 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.797534 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:32:03.797741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.797534 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l7zn9\"" Sep 11 18:32:03.797741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.797538 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-200c07-serving-cert\"" Sep 11 18:32:03.797741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.797629 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 18:32:03.797741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.797544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 18:32:03.804105 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.804081 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz"] Sep 11 18:32:03.824758 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.824697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:03.826828 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.826804 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7828-5713e8"] Sep 11 18:32:03.830299 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.830281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:03.834734 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.834696 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-5713e8"] Sep 11 18:32:03.843173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.843038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-maas-api-tls\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.843173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.843084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjp5\" (UniqueName: \"kubernetes.io/projected/be80f334-e76f-45b8-b3ba-14994c4e9d67-kube-api-access-ghjp5\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.843173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.843113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-metrics-certs\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.843954 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.843192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f-kube-api-access-sf9jl\") pod \"test-curl-7828-5713e8\" (UID: \"d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f\") " pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:03.843954 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.843219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be80f334-e76f-45b8-b3ba-14994c4e9d67-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.925458 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.925416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:03.944174 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.944141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f-kube-api-access-sf9jl\") pod \"test-curl-7828-5713e8\" (UID: \"d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f\") " pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:03.944174 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.944176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be80f334-e76f-45b8-b3ba-14994c4e9d67-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.944401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.944225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-maas-api-tls\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.944401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.944244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjp5\" (UniqueName: \"kubernetes.io/projected/be80f334-e76f-45b8-b3ba-14994c4e9d67-kube-api-access-ghjp5\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.944401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.944262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-metrics-certs\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.944401 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:03.944376 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-serving-cert: secret "maas-api-e2e-worker-w4-200c07-serving-cert" not found Sep 11 18:32:03.944625 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:03.944482 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-maas-api-tls podName:be80f334-e76f-45b8-b3ba-14994c4e9d67 nodeName:}" failed. No retries permitted until 2026-09-11 18:32:04.44446021 +0000 UTC m=+992.671136542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-maas-api-tls") pod "maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" (UID: "be80f334-e76f-45b8-b3ba-14994c4e9d67") : secret "maas-api-e2e-worker-w4-200c07-serving-cert" not found Sep 11 18:32:03.944871 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.944851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be80f334-e76f-45b8-b3ba-14994c4e9d67-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.946882 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.946865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-metrics-certs\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:03.955110 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.955085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f-kube-api-access-sf9jl\") pod \"test-curl-7828-5713e8\" (UID: \"d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f\") " pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:03.955237 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:03.955156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjp5\" (UniqueName: \"kubernetes.io/projected/be80f334-e76f-45b8-b3ba-14994c4e9d67-kube-api-access-ghjp5\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:04.107086 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:04.107055 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7828-83e63d" container="test-curl-7828-83e63d" Sep 11 18:32:04.140558 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.140518 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:04.201263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.201225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:04.417552 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.417526 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-5713e8"] Sep 11 18:32:04.419647 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:32:04.419611 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd72b1d4e_cb9e_4ea7_bfbb_98732354fa6f.slice/crio-d382bbf0ef14a924df0ab1695dea790caf4a5fe2a8833479a307bbdcc050134b WatchSource:0}: Error finding container d382bbf0ef14a924df0ab1695dea790caf4a5fe2a8833479a307bbdcc050134b: Status 404 returned error can't find the container with id d382bbf0ef14a924df0ab1695dea790caf4a5fe2a8833479a307bbdcc050134b Sep 11 18:32:04.422795 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.422767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:04.449935 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.449905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-maas-api-tls\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:04.452315 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.452288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-maas-api-tls\") pod \"maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:04.515681 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.515652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:04.707589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.707509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:04.914366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.914332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:04.976105 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.976016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:04.982067 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:04.982040 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz"] Sep 11 18:32:04.994378 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:32:04.994342 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe80f334_e76f_45b8_b3ba_14994c4e9d67.slice/crio-87e444acf0c9b190dc16fda5edc083e741e18d533f1ec25c33ccaae542676b08 WatchSource:0}: Error finding container 87e444acf0c9b190dc16fda5edc083e741e18d533f1ec25c33ccaae542676b08: Status 404 returned error can't find the container with id 87e444acf0c9b190dc16fda5edc083e741e18d533f1ec25c33ccaae542676b08 Sep 11 18:32:05.114254 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.114203 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-5713e8" event={"ID":"d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f","Type":"ContainerStarted","Data":"506dc7b3240d5948766e2238076a9833dd18beab80a404453eb0109650801220"} Sep 11 18:32:05.114254 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.114252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-5713e8" event={"ID":"d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f","Type":"ContainerStarted","Data":"d382bbf0ef14a924df0ab1695dea790caf4a5fe2a8833479a307bbdcc050134b"} Sep 11 18:32:05.115536 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.115506 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" event={"ID":"be80f334-e76f-45b8-b3ba-14994c4e9d67","Type":"ContainerStarted","Data":"87e444acf0c9b190dc16fda5edc083e741e18d533f1ec25c33ccaae542676b08"} Sep 11 18:32:05.115645 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.115568 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7828-83e63d" podUID="f8419743-e2a3-4221-8aac-018e29ca1ca2" containerName="test-curl-7828-83e63d" containerID="cri-o://d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6" gracePeriod=2 Sep 11 18:32:05.129344 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.129316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:05.139473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.139431 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7828-5713e8" podStartSLOduration=2.139417764 podStartE2EDuration="2.139417764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:05.136322065 +0000 UTC m=+993.362998405" watchObservedRunningTime="2026-09-11 18:32:05.139417764 +0000 UTC m=+993.366094100" Sep 11 18:32:05.387935 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.387891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:05.488450 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.488415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:05.775670 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.775630 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7828-5713e8"] Sep 11 18:32:05.781567 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.781534 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7828-5713e8"] Sep 11 18:32:05.825279 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:05.825242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:06.121073 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:06.120999 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7828-5713e8" container="test-curl-7828-5713e8" Sep 11 18:32:06.297987 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.297947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:06.581097 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.581047 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4"] Sep 11 18:32:06.581960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.581934 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" containerName="test-curl-7828-5713e8" Sep 11 18:32:06.582106 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.582093 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" containerName="test-curl-7828-5713e8" Sep 11 18:32:06.582335 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.582311 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" containerName="test-curl-7828-5713e8" Sep 11 18:32:06.594726 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.594334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.596314 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.595316 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4"] Sep 11 18:32:06.598766 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.598327 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 18:32:06.598766 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.598597 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-536a4d-openshift-default-dockercfg-trk5p\"" Sep 11 18:32:06.605491 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.605469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:06.671062 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.671030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istiod-ca-cert\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.671250 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.671071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-envoy\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.671250 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.671165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-socket\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.671250 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.671215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-credential-socket\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.671425 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.671292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-data\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.671425 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.671348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-certs\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.671425 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.671374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-podinfo\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.671591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.671517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbrt\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-kube-api-access-dwbrt\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.671651 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.671596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-token\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.729077 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.729033 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:06.772655 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.772618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-socket\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.772864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.772668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-credential-socket\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.772864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.772696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-data\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.772864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.772733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-certs\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.772864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.772754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-podinfo\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.772864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.772817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbrt\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-kube-api-access-dwbrt\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.772864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.772840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-token\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.773196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.772869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istiod-ca-cert\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.773196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.772893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-envoy\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.773196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.773075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-socket\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.773196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.773110 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-data\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.773196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.773183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-credential-socket\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.773196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.773193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-certs\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.773829 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.773805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istiod-ca-cert\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.775284 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.775262 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-envoy\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.775872 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.775851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-podinfo\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.781673 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.781646 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-token\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.782231 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.782208 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbrt\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-kube-api-access-dwbrt\") pod \"e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:06.862444 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.862346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:06.909621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:06.909585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:07.128386 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.128356 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7828-5713e8" podUID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" containerName="test-curl-7828-5713e8" containerID="cri-o://506dc7b3240d5948766e2238076a9833dd18beab80a404453eb0109650801220" gracePeriod=2 Sep 11 18:32:07.188499 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.188452 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4"] Sep 11 18:32:07.394360 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.394266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:07.439733 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:32:07.439683 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23deb4f6_f0dd_4f24_943c_b98a903d5b39.slice/crio-fdd43a47e4f74bba3780d8a9cae4c8cbf156794bce162e067977e99717719930 WatchSource:0}: Error finding container fdd43a47e4f74bba3780d8a9cae4c8cbf156794bce162e067977e99717719930: Status 404 returned error can't find the container with id fdd43a47e4f74bba3780d8a9cae4c8cbf156794bce162e067977e99717719930 Sep 11 18:32:07.494749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.494695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:07.569901 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.569874 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-83e63d" Sep 11 18:32:07.595028 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.594986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:07.681358 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.681271 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5pq6\" (UniqueName: \"kubernetes.io/projected/f8419743-e2a3-4221-8aac-018e29ca1ca2-kube-api-access-g5pq6\") pod \"f8419743-e2a3-4221-8aac-018e29ca1ca2\" (UID: \"f8419743-e2a3-4221-8aac-018e29ca1ca2\") " Sep 11 18:32:07.683640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.683606 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8419743-e2a3-4221-8aac-018e29ca1ca2-kube-api-access-g5pq6" (OuterVolumeSpecName: "kube-api-access-g5pq6") pod "f8419743-e2a3-4221-8aac-018e29ca1ca2" (UID: "f8419743-e2a3-4221-8aac-018e29ca1ca2"). InnerVolumeSpecName "kube-api-access-g5pq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:32:07.696191 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.696161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:07.782741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.782362 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5pq6\" (UniqueName: \"kubernetes.io/projected/f8419743-e2a3-4221-8aac-018e29ca1ca2-kube-api-access-g5pq6\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:32:07.947873 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:07.947760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:08.045071 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.045038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:08.135246 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.135215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" event={"ID":"be80f334-e76f-45b8-b3ba-14994c4e9d67","Type":"ContainerStarted","Data":"82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d"} Sep 11 18:32:08.135696 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.135423 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:08.136548 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.136522 2579 generic.go:356] "Generic (PLEG): container finished" podID="f8419743-e2a3-4221-8aac-018e29ca1ca2" containerID="d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6" exitCode=137 Sep 11 18:32:08.136548 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.136555 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-83e63d" Sep 11 18:32:08.136773 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.136609 2579 scope.go:117] "RemoveContainer" containerID="d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6" Sep 11 18:32:08.138161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.138139 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" event={"ID":"23deb4f6-f0dd-4f24-943c-b98a903d5b39","Type":"ContainerStarted","Data":"fdd43a47e4f74bba3780d8a9cae4c8cbf156794bce162e067977e99717719930"} Sep 11 18:32:08.146392 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.146373 2579 scope.go:117] "RemoveContainer" containerID="d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6" Sep 11 18:32:08.146657 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:08.146640 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6\": container with ID starting with d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6 not found: ID does not exist" containerID="d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6" Sep 11 18:32:08.146741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.146663 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6"} err="failed to get container status \"d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6\": rpc error: code = NotFound desc = could not find container \"d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6\": container with ID starting with d3444dfd2c8e95aaec8fcd2880490340cd92346d120bd160ec6177d17e899ef6 not found: ID does not exist" Sep 11 18:32:08.154212 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:08.154176 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-83e63d\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="f8419743-e2a3-4221-8aac-018e29ca1ca2" pod="opendatahub/test-curl-7828-83e63d" Sep 11 18:32:08.155312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.155276 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" podStartSLOduration=2.648970657 podStartE2EDuration="5.155263122s" totalImagesPullingTime="2.506292465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:08.152178974 +0000 UTC m=+996.378855334" watchObservedRunningTime="2026-09-11 18:32:08.155263122 +0000 UTC m=+996.381939462" Sep 11 18:32:08.231928 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:08.231844 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8419743-e2a3-4221-8aac-018e29ca1ca2" path="/var/lib/kubelet/pods/f8419743-e2a3-4221-8aac-018e29ca1ca2/volumes" Sep 11 18:32:09.331734 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:09.331384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:10.090061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.090016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:10.156029 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.155990 2579 generic.go:356] "Generic (PLEG): container finished" podID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" containerID="506dc7b3240d5948766e2238076a9833dd18beab80a404453eb0109650801220" exitCode=137 Sep 11 18:32:10.169461 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.169424 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:10.169591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.169511 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:10.169591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.169555 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:10.213318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.213285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:10.281160 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.281122 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:10.309839 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.309802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f-kube-api-access-sf9jl\") pod \"d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f\" (UID: \"d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f\") " Sep 11 18:32:10.312217 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.312181 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f-kube-api-access-sf9jl" (OuterVolumeSpecName: "kube-api-access-sf9jl") pod "d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" (UID: "d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f"). InnerVolumeSpecName "kube-api-access-sf9jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:32:10.411127 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:10.411090 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f-kube-api-access-sf9jl\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:32:11.102482 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:11.102445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:11.163381 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:11.163351 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:11.163381 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:11.163381 2579 scope.go:117] "RemoveContainer" containerID="506dc7b3240d5948766e2238076a9833dd18beab80a404453eb0109650801220" Sep 11 18:32:11.165415 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:11.165378 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" event={"ID":"23deb4f6-f0dd-4f24-943c-b98a903d5b39","Type":"ContainerStarted","Data":"56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59"} Sep 11 18:32:11.188655 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:11.188610 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" podStartSLOduration=2.461588836 podStartE2EDuration="5.188592858s" totalImagesPullingTime="2.727004022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:11.18704046 +0000 UTC m=+999.413716828" watchObservedRunningTime="2026-09-11 18:32:11.188592858 +0000 UTC m=+999.415269197" Sep 11 18:32:11.189146 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:11.189108 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-5713e8\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:11.910749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:11.910686 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:11.915973 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:11.915947 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:11.936666 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:11.936630 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-5713e8\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:12.011087 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:12.011043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:12.172516 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:12.172434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:12.173311 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:12.173286 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:12.226968 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:12.226933 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-5713e8\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:12.231784 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:12.231758 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" path="/var/lib/kubelet/pods/d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f/volumes" Sep 11 18:32:12.241311 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:12.241275 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-5713e8\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="d72b1d4e-cb9e-4ea7-bfbb-98732354fa6f" pod="opendatahub/test-curl-7828-5713e8" Sep 11 18:32:12.814711 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:12.814661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:13.341994 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.341958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:13.440680 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.440641 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:13.594898 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.594824 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb"] Sep 11 18:32:13.599621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.599599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.602141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.602115 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d6df0e9a-serving-cert\"" Sep 11 18:32:13.612172 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.612147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb"] Sep 11 18:32:13.637332 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.637299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:13.641402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.641379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-metrics-certs\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.641487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.641415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-openshift-service-ca\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.641487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.641477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxsvw\" (UniqueName: \"kubernetes.io/projected/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-kube-api-access-bxsvw\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.641571 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.641545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-maas-api-tls\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.700830 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.700790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:13.742078 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.742041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxsvw\" (UniqueName: \"kubernetes.io/projected/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-kube-api-access-bxsvw\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.742248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.742133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-maas-api-tls\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.742248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.742195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-metrics-certs\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.742337 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.742254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-openshift-service-ca\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.742337 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:13.742291 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-serving-cert: secret "maas-api-e2e-ait-d6df0e9a-serving-cert" not found Sep 11 18:32:13.742417 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:13.742355 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-maas-api-tls podName:abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4 nodeName:}" failed. No retries permitted until 2026-09-11 18:32:14.242334083 +0000 UTC m=+1002.469010418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-maas-api-tls") pod "maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" (UID: "abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4") : secret "maas-api-e2e-ait-d6df0e9a-serving-cert" not found Sep 11 18:32:13.742986 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.742963 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-openshift-service-ca\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.744797 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.744776 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-metrics-certs\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:13.750634 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:13.750609 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxsvw\" (UniqueName: \"kubernetes.io/projected/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-kube-api-access-bxsvw\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:14.039994 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.039956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:14.151828 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.151797 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:32:14.217479 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.217441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:14.246580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.246498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-maas-api-tls\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:14.249635 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.249596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-maas-api-tls\") pod \"maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:14.313017 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.312897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:14.511108 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.511070 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:14.629765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.629728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:14.787062 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.787034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb"] Sep 11 18:32:14.789217 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:32:14.789185 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf58ea4_2efb_41b1_b7a4_1ba9a197c4e4.slice/crio-abfc65fea5a194635bbcca94f00cd4c4f80fc4b930ad2e939984055fd9835067 WatchSource:0}: Error finding container abfc65fea5a194635bbcca94f00cd4c4f80fc4b930ad2e939984055fd9835067: Status 404 returned error can't find the container with id abfc65fea5a194635bbcca94f00cd4c4f80fc4b930ad2e939984055fd9835067 Sep 11 18:32:14.790823 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:14.790803 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:32:15.197475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:15.197439 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" event={"ID":"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4","Type":"ContainerStarted","Data":"abfc65fea5a194635bbcca94f00cd4c4f80fc4b930ad2e939984055fd9835067"} Sep 11 18:32:15.697998 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:15.697942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:15.771932 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:15.771899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:15.889190 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:15.889135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:16.205738 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.205691 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" event={"ID":"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4","Type":"ContainerStarted","Data":"5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc"} Sep 11 18:32:16.205942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.205831 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:16.206029 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.206005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:16.226957 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.226906 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" podStartSLOduration=2.797274416 podStartE2EDuration="3.226889746s" totalImagesPullingTime="429.61533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:16.224483875 +0000 UTC m=+1004.451160239" watchObservedRunningTime="2026-09-11 18:32:16.226889746 +0000 UTC m=+1004.453566085" Sep 11 18:32:16.300453 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.300407 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:16.388603 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.388211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:16.629629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.629590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:16.751248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.751207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:16.850478 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.850441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:16.946727 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.946618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:16.948379 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.948357 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4"] Sep 11 18:32:16.952081 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.952055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:16.954746 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.954726 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e8f201-openshift-default-dockercfg-h6qlp\"" Sep 11 18:32:16.963449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:16.963427 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4"] Sep 11 18:32:17.007349 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.007313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:17.076542 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.076498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-socket\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.076741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.076565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-envoy\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.076741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.076610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-credential-socket\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.076741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.076677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f015fd-0119-494c-96bc-8726b34166ad-istiod-ca-cert\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.076953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.076755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-data\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.076953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.076789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f015fd-0119-494c-96bc-8726b34166ad-istio-podinfo\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.076953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.076819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-istio-token\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.076953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.076883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-certs\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.076953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.076902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldlx\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-kube-api-access-wldlx\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.178420 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-certs\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.178420 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldlx\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-kube-api-access-wldlx\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.178661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-socket\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.178661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-envoy\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.178661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-credential-socket\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.178661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f015fd-0119-494c-96bc-8726b34166ad-istiod-ca-cert\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.178661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-data\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.178661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f015fd-0119-494c-96bc-8726b34166ad-istio-podinfo\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.178661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-istio-token\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.179051 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.178953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-certs\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.179051 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.179025 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-socket\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.179162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.179092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-data\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.179219 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.179202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-credential-socket\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.179337 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.179317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f015fd-0119-494c-96bc-8726b34166ad-istiod-ca-cert\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.181065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.181037 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-envoy\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.181218 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.181204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f015fd-0119-494c-96bc-8726b34166ad-istio-podinfo\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.200041 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.199967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-istio-token\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.205046 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.205020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldlx\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-kube-api-access-wldlx\") pod \"e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.263892 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.263860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:17.420397 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.420362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:17.546151 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.546121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4"] Sep 11 18:32:17.547007 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:32:17.546979 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f015fd_0119_494c_96bc_8726b34166ad.slice/crio-6c05637af05f80ae6809ef656d6b00969590085748e97b82ac8733ea64451e40 WatchSource:0}: Error finding container 6c05637af05f80ae6809ef656d6b00969590085748e97b82ac8733ea64451e40: Status 404 returned error can't find the container with id 6c05637af05f80ae6809ef656d6b00969590085748e97b82ac8733ea64451e40 Sep 11 18:32:17.550401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.550367 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:17.550483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.550432 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:17.550483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.550462 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:17.750763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:17.750725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:18.220278 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:18.220240 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" event={"ID":"82f015fd-0119-494c-96bc-8726b34166ad","Type":"ContainerStarted","Data":"32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597"} Sep 11 18:32:18.220734 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:18.220301 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" event={"ID":"82f015fd-0119-494c-96bc-8726b34166ad","Type":"ContainerStarted","Data":"6c05637af05f80ae6809ef656d6b00969590085748e97b82ac8733ea64451e40"} Sep 11 18:32:18.241714 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:18.241661 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" podStartSLOduration=2.241646006 podStartE2EDuration="2.241646006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:18.239351443 +0000 UTC m=+1006.466027804" watchObservedRunningTime="2026-09-11 18:32:18.241646006 +0000 UTC m=+1006.468322345" Sep 11 18:32:18.264351 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:18.264322 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:18.269212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:18.269188 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:18.621876 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:18.621843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:18.908533 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:18.908434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:19.067514 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:19.067481 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:19.179722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:19.179612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:19.226607 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:19.226572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:19.227501 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:19.227478 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:19.485039 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:19.484955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:19.719970 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:19.719935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:20.232079 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:20.232039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:20.324966 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:20.324929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:20.446367 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:20.446330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:20.755149 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:20.755110 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:32:21.341859 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:21.341827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:21.506987 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:21.506949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:21.980566 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:21.980529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:22.069115 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:22.069084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:22.168823 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:22.168788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:22.218199 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:22.218163 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:22.269457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:22.269417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:23.139197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:23.139151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:23.966665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:23.966628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:24.087983 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.087948 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh"] Sep 11 18:32:24.106029 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.105991 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.125740 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.122127 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e8f201-serving-cert\"" Sep 11 18:32:24.131117 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.131019 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh"] Sep 11 18:32:24.142164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.142108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:24.243069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.242970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:24.254430 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.254397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-maas-api-tls\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.254597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.254528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b16857b-a659-4549-99b8-74f9cf1f9353-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.254597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.254566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-metrics-certs\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.254691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.254597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcdjz\" (UniqueName: \"kubernetes.io/projected/2b16857b-a659-4549-99b8-74f9cf1f9353-kube-api-access-jcdjz\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.342810 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.342776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:24.355299 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.355270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdjz\" (UniqueName: \"kubernetes.io/projected/2b16857b-a659-4549-99b8-74f9cf1f9353-kube-api-access-jcdjz\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.355715 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.355672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-maas-api-tls\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.355972 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.355947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b16857b-a659-4549-99b8-74f9cf1f9353-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.356182 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.356160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-metrics-certs\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.356533 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.356504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b16857b-a659-4549-99b8-74f9cf1f9353-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.358157 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.358136 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-maas-api-tls\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.358454 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.358436 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-metrics-certs\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.363744 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.363722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdjz\" (UniqueName: \"kubernetes.io/projected/2b16857b-a659-4549-99b8-74f9cf1f9353-kube-api-access-jcdjz\") pod \"maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.440397 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.440359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:24.607752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.607675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:24.701743 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.698601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:24.745930 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:24.745894 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh"] Sep 11 18:32:24.748760 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:32:24.748729 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b16857b_a659_4549_99b8_74f9cf1f9353.slice/crio-80e968b7eb2f501fbfaaa50911185fcfd86b04ff10f96bb1b1c726bb7415ff68 WatchSource:0}: Error finding container 80e968b7eb2f501fbfaaa50911185fcfd86b04ff10f96bb1b1c726bb7415ff68: Status 404 returned error can't find the container with id 80e968b7eb2f501fbfaaa50911185fcfd86b04ff10f96bb1b1c726bb7415ff68 Sep 11 18:32:25.111350 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:25.111302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:25.264255 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:25.264221 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" event={"ID":"2b16857b-a659-4549-99b8-74f9cf1f9353","Type":"ContainerStarted","Data":"80e968b7eb2f501fbfaaa50911185fcfd86b04ff10f96bb1b1c726bb7415ff68"} Sep 11 18:32:25.771881 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:25.771850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:26.271443 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:26.271402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" event={"ID":"2b16857b-a659-4549-99b8-74f9cf1f9353","Type":"ContainerStarted","Data":"cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1"} Sep 11 18:32:26.271855 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:26.271469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:26.288476 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:26.288405 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" podStartSLOduration=1.6826196850000001 podStartE2EDuration="2.288384626s" totalImagesPullingTime="605.764941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:26.288144614 +0000 UTC m=+1014.514820973" watchObservedRunningTime="2026-09-11 18:32:26.288384626 +0000 UTC m=+1014.515060968" Sep 11 18:32:26.712425 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:26.712340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:27.413173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:27.413139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:28.175816 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:28.175767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:28.275797 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:28.275763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:28.394192 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:28.394146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:31.305508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:31.305475 2579 generic.go:356] "Generic (PLEG): container finished" podID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerID="56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59" exitCode=0 Sep 11 18:32:31.305984 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:31.305554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" event={"ID":"23deb4f6-f0dd-4f24-943c-b98a903d5b39","Type":"ContainerDied","Data":"56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59"} Sep 11 18:32:31.306089 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:31.306070 2579 scope.go:117] "RemoveContainer" containerID="56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59" Sep 11 18:32:31.308184 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:31.308151 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:31.308312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:31.308221 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:31.308312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:31.308277 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:32.284082 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:32.284052 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:32:32.314727 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:32.314678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" event={"ID":"23deb4f6-f0dd-4f24-943c-b98a903d5b39","Type":"ContainerStarted","Data":"e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546"} Sep 11 18:32:32.910364 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:32.910321 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:32.910608 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:32.910577 2579 patch_prober.go:28] interesting pod/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:32:32.910690 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:32.910661 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:32:33.910615 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:33.910573 2579 patch_prober.go:28] interesting pod/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:32:33.911204 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:33.911169 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:32:34.313744 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:34.313680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:34.420236 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:34.420193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:34.511616 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:34.511579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:34.681473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:34.681383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:34.778140 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:34.778099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:34.910699 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:34.910655 2579 patch_prober.go:28] interesting pod/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:32:34.911134 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:34.910739 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:32:35.910742 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:35.910679 2579 patch_prober.go:28] interesting pod/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:32:35.911247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:35.910769 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:32:36.123429 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:36.123384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:36.883023 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:36.882987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:36.910994 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:36.910958 2579 patch_prober.go:28] interesting pod/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:32:36.911552 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:36.911523 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:32:37.712944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:37.712906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:37.910300 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:37.910262 2579 patch_prober.go:28] interesting pod/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:32:37.910574 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:37.910545 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:32:38.350056 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:38.350017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:38.915426 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:38.915387 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:38.915900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:38.915872 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:38.916936 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:38.916914 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:32:39.015853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:39.015812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:39.794640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:39.794565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:40.611084 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:40.611027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:41.089122 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:41.089086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:41.178218 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:41.178178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:41.315696 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:41.315604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:42.103771 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:42.103729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:42.442281 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:42.442191 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:32:42.870877 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:42.870844 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:43.221671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:43.221565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:43.608958 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:43.608913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:44.287584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:44.287548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:44.395233 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:44.395158 2579 generic.go:356] "Generic (PLEG): container finished" podID="82f015fd-0119-494c-96bc-8726b34166ad" containerID="32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597" exitCode=0 Sep 11 18:32:44.395404 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:44.395239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" event={"ID":"82f015fd-0119-494c-96bc-8726b34166ad","Type":"ContainerDied","Data":"32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597"} Sep 11 18:32:44.395757 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:44.395739 2579 scope.go:117] "RemoveContainer" containerID="32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597" Sep 11 18:32:44.397075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:44.397048 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:44.397137 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:44.397120 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:44.397174 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:44.397154 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:32:45.010630 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:45.010594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:45.110570 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:45.110531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:45.410196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:45.410081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" event={"ID":"82f015fd-0119-494c-96bc-8726b34166ad","Type":"ContainerStarted","Data":"45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068"} Sep 11 18:32:45.703840 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:45.703738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:46.264849 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:46.264801 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:46.266057 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:46.266029 2579 patch_prober.go:28] interesting pod/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 18:32:46.266141 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:46.266082 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 18:32:47.264985 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:47.264951 2579 patch_prober.go:28] interesting pod/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 18:32:47.265390 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:47.265004 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 18:32:48.264555 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:48.264505 2579 patch_prober.go:28] interesting pod/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 18:32:48.264797 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:48.264579 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 18:32:49.227027 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:49.226982 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:49.264803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:49.264768 2579 patch_prober.go:28] interesting pod/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 18:32:49.264932 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:49.264825 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 18:32:50.264554 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:50.264510 2579 patch_prober.go:28] interesting pod/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 18:32:50.264972 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:50.264574 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 18:32:51.268682 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:51.268649 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:51.269864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:51.269840 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:32:51.804162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:51.804122 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb"] Sep 11 18:32:51.804511 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:51.804471 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" podUID="abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" containerName="maas-api" containerID="cri-o://5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc" gracePeriod=30 Sep 11 18:32:51.832880 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:51.832855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:51.899793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:51.899758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:52.000601 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.000568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:52.056275 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.056209 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:52.078413 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.078370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:52.152917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.152440 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-maas-api-tls\") pod \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " Sep 11 18:32:52.152917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.152572 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-openshift-service-ca\") pod \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " Sep 11 18:32:52.152917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.152647 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxsvw\" (UniqueName: \"kubernetes.io/projected/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-kube-api-access-bxsvw\") pod \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " Sep 11 18:32:52.152917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.152682 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-metrics-certs\") pod \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\" (UID: \"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4\") " Sep 11 18:32:52.153912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.153876 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" (UID: "abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:32:52.157568 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.157520 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" (UID: "abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:32:52.159096 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.159052 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" (UID: "abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:32:52.159816 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.159786 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-kube-api-access-bxsvw" (OuterVolumeSpecName: "kube-api-access-bxsvw") pod "abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" (UID: "abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4"). InnerVolumeSpecName "kube-api-access-bxsvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:32:52.172391 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.172358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:52.255031 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.254996 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:32:52.255031 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.255025 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxsvw\" (UniqueName: \"kubernetes.io/projected/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-kube-api-access-bxsvw\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:32:52.255226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.255039 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:32:52.255226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.255051 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:32:52.453281 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.453166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:52.453762 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.453311 2579 generic.go:356] "Generic (PLEG): container finished" podID="abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" containerID="5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc" exitCode=0 Sep 11 18:32:52.453762 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.453350 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" Sep 11 18:32:52.453762 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.453385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" event={"ID":"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4","Type":"ContainerDied","Data":"5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc"} Sep 11 18:32:52.453762 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.453446 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb" event={"ID":"abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4","Type":"ContainerDied","Data":"abfc65fea5a194635bbcca94f00cd4c4f80fc4b930ad2e939984055fd9835067"} Sep 11 18:32:52.453762 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.453479 2579 scope.go:117] "RemoveContainer" containerID="5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc" Sep 11 18:32:52.463047 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.462993 2579 scope.go:117] "RemoveContainer" containerID="5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc" Sep 11 18:32:52.463288 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:52.463267 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc\": container with ID starting with 5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc not found: ID does not exist" containerID="5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc" Sep 11 18:32:52.463351 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.463294 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc"} err="failed to get container status \"5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc\": rpc error: code = NotFound desc = could not find container \"5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc\": container with ID starting with 5ef2a16d5782e78690a8ac930b7ad07f873593ee7b33a0a2706401c220eed0cc not found: ID does not exist" Sep 11 18:32:52.476716 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.476673 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb"] Sep 11 18:32:52.479791 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.479765 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6df0e9a-754b8d5dcb-bnxbb"] Sep 11 18:32:52.641953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.641917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:52.740349 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.740255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:52.867369 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:52.867327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:53.203423 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:53.203387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:53.343811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:53.343770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:53.444300 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:53.444268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:54.050984 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:54.050944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:54.229338 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:54.229307 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" path="/var/lib/kubelet/pods/abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4/volumes" Sep 11 18:32:54.524549 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:54.524494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:54.627756 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:54.627697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:54.947960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:54.947867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:55.701029 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:55.700994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:55.801997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:55.801963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:55.905475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:55.905441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:56.810288 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:56.810246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:57.503153 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.503120 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw"] Sep 11 18:32:57.503803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.503768 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" containerName="maas-api" Sep 11 18:32:57.503803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.503791 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" containerName="maas-api" Sep 11 18:32:57.503978 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.503863 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf58ea4-2efb-41b1-b7a4-1ba9a197c4e4" containerName="maas-api" Sep 11 18:32:57.506793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.506773 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.509763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.509742 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-414d08da-serving-cert\"" Sep 11 18:32:57.518493 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.518472 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw"] Sep 11 18:32:57.539564 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.539536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:57.565500 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.565466 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:32:57.606567 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.606526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a85815-ba55-4e96-8a05-7d369cb96743-openshift-service-ca\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.606803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.606632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-maas-api-tls\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.606803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.606685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-metrics-certs\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.606898 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.606860 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrgr\" (UniqueName: \"kubernetes.io/projected/a0a85815-ba55-4e96-8a05-7d369cb96743-kube-api-access-frrgr\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.639347 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.639308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:57.707761 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.707690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-metrics-certs\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.707933 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.707820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrgr\" (UniqueName: \"kubernetes.io/projected/a0a85815-ba55-4e96-8a05-7d369cb96743-kube-api-access-frrgr\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.707933 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.707872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a85815-ba55-4e96-8a05-7d369cb96743-openshift-service-ca\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.708071 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.707942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-maas-api-tls\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.708134 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:57.708069 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-serving-cert: secret "maas-api-e2e-migrate-414d08da-serving-cert" not found Sep 11 18:32:57.708184 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:32:57.708139 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-maas-api-tls podName:a0a85815-ba55-4e96-8a05-7d369cb96743 nodeName:}" failed. No retries permitted until 2026-09-11 18:32:58.208118899 +0000 UTC m=+1046.434795233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-maas-api-tls") pod "maas-api-e2e-migrate-414d08da-5b47646768-r59vw" (UID: "a0a85815-ba55-4e96-8a05-7d369cb96743") : secret "maas-api-e2e-migrate-414d08da-serving-cert" not found Sep 11 18:32:57.708654 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.708627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a85815-ba55-4e96-8a05-7d369cb96743-openshift-service-ca\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.710480 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.710458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-metrics-certs\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.717622 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.717597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrgr\" (UniqueName: \"kubernetes.io/projected/a0a85815-ba55-4e96-8a05-7d369cb96743-kube-api-access-frrgr\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:57.846674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:57.846554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:58.215040 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:58.214959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-maas-api-tls\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:58.217612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:58.217586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-maas-api-tls\") pod \"maas-api-e2e-migrate-414d08da-5b47646768-r59vw\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:58.239504 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:58.239464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:58.344401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:58.344373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:58.418524 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:58.418483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:58.704904 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:58.704877 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw"] Sep 11 18:32:58.705808 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:32:58.705783 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a85815_ba55_4e96_8a05_7d369cb96743.slice/crio-f329e778a0e11bc64241d266d734c5958db30044d29b2e74978015b6f7e12b54 WatchSource:0}: Error finding container f329e778a0e11bc64241d266d734c5958db30044d29b2e74978015b6f7e12b54: Status 404 returned error can't find the container with id f329e778a0e11bc64241d266d734c5958db30044d29b2e74978015b6f7e12b54 Sep 11 18:32:58.808878 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:58.808841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:59.496164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:59.496069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:59.513463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:59.513429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" event={"ID":"a0a85815-ba55-4e96-8a05-7d369cb96743","Type":"ContainerStarted","Data":"0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48"} Sep 11 18:32:59.513636 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:59.513485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" event={"ID":"a0a85815-ba55-4e96-8a05-7d369cb96743","Type":"ContainerStarted","Data":"f329e778a0e11bc64241d266d734c5958db30044d29b2e74978015b6f7e12b54"} Sep 11 18:32:59.513636 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:59.513619 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:32:59.530640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:32:59.530592 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" podStartSLOduration=2.075824498 podStartE2EDuration="2.530576519s" totalImagesPullingTime="454.752021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:59.529226742 +0000 UTC m=+1047.755903101" watchObservedRunningTime="2026-09-11 18:32:59.530576519 +0000 UTC m=+1047.757252858" Sep 11 18:33:00.141185 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:00.141152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:00.920041 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:00.920003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:05.524131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:05.524101 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:33:12.340119 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:12.340082 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:33:14.048049 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:14.048008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:16.550876 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:16.550842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:18.806071 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:18.806035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:20.028690 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:20.028642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:20.898962 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:20.898891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:21.548061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:21.548027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:22.316880 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:22.316833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:23.213318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:23.213282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:23.901698 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:23.901665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:24.448918 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:24.448881 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:33:24.736470 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:24.736358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:25.504257 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:25.504223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:26.313044 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:26.313004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:26.856811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:26.856775 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw"] Sep 11 18:33:26.857247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:26.857032 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" podUID="a0a85815-ba55-4e96-8a05-7d369cb96743" containerName="maas-api" containerID="cri-o://0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48" gracePeriod=30 Sep 11 18:33:26.886767 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:26.886733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:26.957847 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:26.957812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:27.015074 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.015033 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:27.079242 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.079209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:27.125640 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.125617 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:33:27.132578 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.132514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:27.205839 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.205805 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a85815-ba55-4e96-8a05-7d369cb96743-openshift-service-ca\") pod \"a0a85815-ba55-4e96-8a05-7d369cb96743\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " Sep 11 18:33:27.206002 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.205883 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-maas-api-tls\") pod \"a0a85815-ba55-4e96-8a05-7d369cb96743\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " Sep 11 18:33:27.206002 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.205943 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frrgr\" (UniqueName: \"kubernetes.io/projected/a0a85815-ba55-4e96-8a05-7d369cb96743-kube-api-access-frrgr\") pod \"a0a85815-ba55-4e96-8a05-7d369cb96743\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " Sep 11 18:33:27.206126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.206003 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-metrics-certs\") pod \"a0a85815-ba55-4e96-8a05-7d369cb96743\" (UID: \"a0a85815-ba55-4e96-8a05-7d369cb96743\") " Sep 11 18:33:27.206263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.206234 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a85815-ba55-4e96-8a05-7d369cb96743-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0a85815-ba55-4e96-8a05-7d369cb96743" (UID: "a0a85815-ba55-4e96-8a05-7d369cb96743"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:33:27.208850 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.208810 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a85815-ba55-4e96-8a05-7d369cb96743-kube-api-access-frrgr" (OuterVolumeSpecName: "kube-api-access-frrgr") pod "a0a85815-ba55-4e96-8a05-7d369cb96743" (UID: "a0a85815-ba55-4e96-8a05-7d369cb96743"). InnerVolumeSpecName "kube-api-access-frrgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:33:27.208850 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.208810 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a0a85815-ba55-4e96-8a05-7d369cb96743" (UID: "a0a85815-ba55-4e96-8a05-7d369cb96743"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:33:27.208850 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.208827 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0a85815-ba55-4e96-8a05-7d369cb96743" (UID: "a0a85815-ba55-4e96-8a05-7d369cb96743"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:33:27.224865 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.224834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:27.307379 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.307339 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:33:27.307379 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.307375 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a85815-ba55-4e96-8a05-7d369cb96743-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:33:27.307606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.307523 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a85815-ba55-4e96-8a05-7d369cb96743-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:33:27.307606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.307533 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frrgr\" (UniqueName: \"kubernetes.io/projected/a0a85815-ba55-4e96-8a05-7d369cb96743-kube-api-access-frrgr\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:33:27.421604 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.421511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:27.653988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.653954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:27.708923 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.708824 2579 generic.go:356] "Generic (PLEG): container finished" podID="a0a85815-ba55-4e96-8a05-7d369cb96743" containerID="0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48" exitCode=0 Sep 11 18:33:27.708923 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.708881 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" Sep 11 18:33:27.709129 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.708886 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" event={"ID":"a0a85815-ba55-4e96-8a05-7d369cb96743","Type":"ContainerDied","Data":"0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48"} Sep 11 18:33:27.709129 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.709005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw" event={"ID":"a0a85815-ba55-4e96-8a05-7d369cb96743","Type":"ContainerDied","Data":"f329e778a0e11bc64241d266d734c5958db30044d29b2e74978015b6f7e12b54"} Sep 11 18:33:27.709129 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.709029 2579 scope.go:117] "RemoveContainer" containerID="0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48" Sep 11 18:33:27.719500 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.719484 2579 scope.go:117] "RemoveContainer" containerID="0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48" Sep 11 18:33:27.719844 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:33:27.719813 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48\": container with ID starting with 0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48 not found: ID does not exist" containerID="0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48" Sep 11 18:33:27.719937 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.719854 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48"} err="failed to get container status \"0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48\": rpc error: code = NotFound desc = could not find container \"0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48\": container with ID starting with 0358d84ce06b3119b6ad40bd5e9258e1cd161aa46f1f4ffa8bee553b1392dc48 not found: ID does not exist" Sep 11 18:33:27.732514 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.732491 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw"] Sep 11 18:33:27.735904 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.735879 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-414d08da-5b47646768-r59vw"] Sep 11 18:33:27.951286 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:27.951252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:28.091344 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:28.091310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:28.194064 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:28.194029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:28.231075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:28.231041 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a85815-ba55-4e96-8a05-7d369cb96743" path="/var/lib/kubelet/pods/a0a85815-ba55-4e96-8a05-7d369cb96743/volumes" Sep 11 18:33:28.952861 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:28.952824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:29.736632 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:29.736585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:30.510122 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:30.510083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:30.835205 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:30.835116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:30.926144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:30.926106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:31.396403 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:31.396367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:32.131314 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.131277 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh"] Sep 11 18:33:32.131772 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.131747 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a85815-ba55-4e96-8a05-7d369cb96743" containerName="maas-api" Sep 11 18:33:32.131772 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.131773 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a85815-ba55-4e96-8a05-7d369cb96743" containerName="maas-api" Sep 11 18:33:32.131868 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.131850 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a85815-ba55-4e96-8a05-7d369cb96743" containerName="maas-api" Sep 11 18:33:32.135151 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.135125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.138538 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.138517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f8e161a6-serving-cert\"" Sep 11 18:33:32.147910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.147864 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh"] Sep 11 18:33:32.177389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.177358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:32.254734 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.254680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-openshift-service-ca\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.254876 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.254819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-metrics-certs\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.254933 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.254900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-maas-api-tls\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.254933 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.254922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkbd\" (UniqueName: \"kubernetes.io/projected/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-kube-api-access-tfkbd\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.278387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.278347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:32.356346 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.356276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-maas-api-tls\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.356346 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.356316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkbd\" (UniqueName: \"kubernetes.io/projected/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-kube-api-access-tfkbd\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.356467 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.356373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-openshift-service-ca\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.356529 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.356470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-metrics-certs\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.357328 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.357306 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-openshift-service-ca\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.358970 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.358943 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-metrics-certs\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.359177 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.359162 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f8e161a6-serving-cert\"" Sep 11 18:33:32.364863 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.364843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkbd\" (UniqueName: \"kubernetes.io/projected/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-kube-api-access-tfkbd\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.368922 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.368899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-maas-api-tls\") pod \"maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.377361 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.377325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:32.447878 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.447798 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:32.643585 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.643553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:32.725922 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.725893 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh"] Sep 11 18:33:32.727405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.727371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:32.727956 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:33:32.727932 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a40f91b_dff7_4c63_b1fa_b3f6e7e6f246.slice/crio-bd69421b56f3c5e598ac22e33eb217d20496fba9530f946b62c0966a5481880d WatchSource:0}: Error finding container bd69421b56f3c5e598ac22e33eb217d20496fba9530f946b62c0966a5481880d: Status 404 returned error can't find the container with id bd69421b56f3c5e598ac22e33eb217d20496fba9530f946b62c0966a5481880d Sep 11 18:33:32.741366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:32.741342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" event={"ID":"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246","Type":"ContainerStarted","Data":"bd69421b56f3c5e598ac22e33eb217d20496fba9530f946b62c0966a5481880d"} Sep 11 18:33:33.097555 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:33.097519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:33.749086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:33.749046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" event={"ID":"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246","Type":"ContainerStarted","Data":"7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7"} Sep 11 18:33:33.749473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:33.749110 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:33.768106 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:33.768054 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" podStartSLOduration=1.348963055 podStartE2EDuration="1.768037528s" totalImagesPullingTime="419.074473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:33.765156442 +0000 UTC m=+1081.991832783" watchObservedRunningTime="2026-09-11 18:33:33.768037528 +0000 UTC m=+1081.994713879" Sep 11 18:33:33.888088 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:33.888046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:34.687403 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:34.687371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:35.319132 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:35.319091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:35.952907 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:35.952874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:39.762252 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:39.762219 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:33:41.371587 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:41.366168 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:33:44.735920 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:44.735887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:54.201361 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:54.201326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:55.146300 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:55.146263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:55.927986 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:55.927946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:56.959446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:56.959404 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:57.693822 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:57.693783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:58.550654 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:58.550617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:59.393828 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:33:59.393795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:00.122463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:00.122413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:01.047770 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:01.047733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:01.256382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:01.256343 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:34:01.623363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:01.623327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:01.724971 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:01.724930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:01.822905 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:01.822868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:02.481008 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.480970 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh"] Sep 11 18:34:02.481450 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.481243 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" podUID="0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" containerName="maas-api" containerID="cri-o://7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7" gracePeriod=30 Sep 11 18:34:02.514982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.514946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:02.616324 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.616290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:02.715776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.715725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:02.744044 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.743975 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:34:02.819274 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.819232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:02.837503 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.837474 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-openshift-service-ca\") pod \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " Sep 11 18:34:02.837689 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.837544 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-metrics-certs\") pod \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " Sep 11 18:34:02.837689 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.837611 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-maas-api-tls\") pod \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " Sep 11 18:34:02.837689 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.837636 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfkbd\" (UniqueName: \"kubernetes.io/projected/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-kube-api-access-tfkbd\") pod \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\" (UID: \"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246\") " Sep 11 18:34:02.838008 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.837958 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" (UID: "0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:34:02.839845 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.839822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" (UID: "0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:34:02.840159 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.840130 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-kube-api-access-tfkbd" (OuterVolumeSpecName: "kube-api-access-tfkbd") pod "0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" (UID: "0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246"). InnerVolumeSpecName "kube-api-access-tfkbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:34:02.840229 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.840210 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" (UID: "0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:34:02.928656 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.928618 2579 generic.go:356] "Generic (PLEG): container finished" podID="0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" containerID="7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7" exitCode=0 Sep 11 18:34:02.928843 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.928673 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" Sep 11 18:34:02.928843 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.928722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" event={"ID":"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246","Type":"ContainerDied","Data":"7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7"} Sep 11 18:34:02.928843 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.928783 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh" event={"ID":"0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246","Type":"ContainerDied","Data":"bd69421b56f3c5e598ac22e33eb217d20496fba9530f946b62c0966a5481880d"} Sep 11 18:34:02.928843 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.928806 2579 scope.go:117] "RemoveContainer" containerID="7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7" Sep 11 18:34:02.940893 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.940869 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:34:02.940893 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.940893 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:34:02.941104 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.940902 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfkbd\" (UniqueName: \"kubernetes.io/projected/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-kube-api-access-tfkbd\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:34:02.941104 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.940911 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:34:02.942329 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.942301 2579 scope.go:117] "RemoveContainer" containerID="7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7" Sep 11 18:34:02.942690 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:34:02.942666 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7\": container with ID starting with 7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7 not found: ID does not exist" containerID="7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7" Sep 11 18:34:02.942765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.942730 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7"} err="failed to get container status \"7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7\": rpc error: code = NotFound desc = could not find container \"7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7\": container with ID starting with 7e59e1016d8fdb415eae1d1cb7317e94d98d86beeeb161bc32bd273e72fcb8c7 not found: ID does not exist" Sep 11 18:34:02.967402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.967369 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh"] Sep 11 18:34:02.979100 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:02.979068 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8e161a6-58579d95fc-t8bwh"] Sep 11 18:34:03.180746 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:03.180698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:03.875697 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:03.875664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:04.155458 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:04.155367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:04.229639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:04.229604 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" path="/var/lib/kubelet/pods/0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246/volumes" Sep 11 18:34:04.306283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:04.306248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:04.617914 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:04.617879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:04.877599 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:04.877487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:04.980087 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:04.980051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:05.729748 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:05.729684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:06.562408 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:06.562366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:07.141459 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:07.141408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:07.231452 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:07.231411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:07.431194 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:07.431069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:08.398167 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.398135 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc"] Sep 11 18:34:08.398800 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.398785 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" containerName="maas-api" Sep 11 18:34:08.398853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.398804 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" containerName="maas-api" Sep 11 18:34:08.398948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.398937 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a40f91b-dff7-4c63-b1fa-b3f6e7e6f246" containerName="maas-api" Sep 11 18:34:08.403765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.403746 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.406388 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.406361 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0b2b4b09-serving-cert\"" Sep 11 18:34:08.417964 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.417940 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc"] Sep 11 18:34:08.441671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.441646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:08.494966 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.494940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-maas-api-tls\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.495114 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.494975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3680c6-5d25-4767-952a-547657cbd6d0-openshift-service-ca\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.495156 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.495114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6q4\" (UniqueName: \"kubernetes.io/projected/af3680c6-5d25-4767-952a-547657cbd6d0-kube-api-access-6c6q4\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.495191 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.495152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-metrics-certs\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.542803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.542759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:08.596250 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.596214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6q4\" (UniqueName: \"kubernetes.io/projected/af3680c6-5d25-4767-952a-547657cbd6d0-kube-api-access-6c6q4\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.596451 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.596270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-metrics-certs\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.596451 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.596327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-maas-api-tls\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.596451 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.596358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3680c6-5d25-4767-952a-547657cbd6d0-openshift-service-ca\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.597394 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.597370 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3680c6-5d25-4767-952a-547657cbd6d0-openshift-service-ca\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.599495 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.599471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-metrics-certs\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.599639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.599558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-maas-api-tls\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.609567 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.609544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6q4\" (UniqueName: \"kubernetes.io/projected/af3680c6-5d25-4767-952a-547657cbd6d0-kube-api-access-6c6q4\") pod \"maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.642160 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.642129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:08.714806 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.714726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:08.978130 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:08.978053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:09.000176 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:09.000147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc"] Sep 11 18:34:09.001033 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:34:09.000998 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3680c6_5d25_4767_952a_547657cbd6d0.slice/crio-0361a957281d4fe3be89c0b381e964e42581c969c4ffc293b46d131945dc3916 WatchSource:0}: Error finding container 0361a957281d4fe3be89c0b381e964e42581c969c4ffc293b46d131945dc3916: Status 404 returned error can't find the container with id 0361a957281d4fe3be89c0b381e964e42581c969c4ffc293b46d131945dc3916 Sep 11 18:34:09.072959 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:09.072925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:09.428208 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:09.428174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:09.972890 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:09.972854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" event={"ID":"af3680c6-5d25-4767-952a-547657cbd6d0","Type":"ContainerStarted","Data":"a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed"} Sep 11 18:34:09.973080 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:09.972911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" event={"ID":"af3680c6-5d25-4767-952a-547657cbd6d0","Type":"ContainerStarted","Data":"0361a957281d4fe3be89c0b381e964e42581c969c4ffc293b46d131945dc3916"} Sep 11 18:34:09.973080 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:09.972949 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:09.998634 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:09.998583 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" podStartSLOduration=1.492874266 podStartE2EDuration="1.998564813s" totalImagesPullingTime="505.690547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:09.996463896 +0000 UTC m=+1118.223140266" watchObservedRunningTime="2026-09-11 18:34:09.998564813 +0000 UTC m=+1118.225241153" Sep 11 18:34:10.200472 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:10.200362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:11.044467 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:11.044435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:11.794143 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:11.794097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:12.572015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:12.571973 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:15.353897 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:15.353858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:15.985738 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:15.985687 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:20.476496 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:20.476458 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:34:30.813473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:30.813437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:31.567937 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:31.567903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:32.429414 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:32.429382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:33.241665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:33.241625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:34.008795 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:34.008762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:34.798971 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:34.798934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:35.640558 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:35.640525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:36.365638 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:36.365602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:37.078765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:37.078699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:38.549580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:38.549547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:39.200987 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.200956 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc"] Sep 11 18:34:39.201166 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.201144 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" podUID="af3680c6-5d25-4767-952a-547657cbd6d0" containerName="maas-api" containerID="cri-o://a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed" gracePeriod=30 Sep 11 18:34:39.267465 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.267425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:39.396687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.395892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:39.464010 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.463951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:39.471026 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.471001 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:39.508774 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.508743 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-metrics-certs\") pod \"af3680c6-5d25-4767-952a-547657cbd6d0\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " Sep 11 18:34:39.508957 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.508941 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-maas-api-tls\") pod \"af3680c6-5d25-4767-952a-547657cbd6d0\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " Sep 11 18:34:39.509031 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.508998 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3680c6-5d25-4767-952a-547657cbd6d0-openshift-service-ca\") pod \"af3680c6-5d25-4767-952a-547657cbd6d0\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " Sep 11 18:34:39.509090 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.509058 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c6q4\" (UniqueName: \"kubernetes.io/projected/af3680c6-5d25-4767-952a-547657cbd6d0-kube-api-access-6c6q4\") pod \"af3680c6-5d25-4767-952a-547657cbd6d0\" (UID: \"af3680c6-5d25-4767-952a-547657cbd6d0\") " Sep 11 18:34:39.509466 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.509433 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3680c6-5d25-4767-952a-547657cbd6d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af3680c6-5d25-4767-952a-547657cbd6d0" (UID: "af3680c6-5d25-4767-952a-547657cbd6d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:34:39.511156 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.511131 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3680c6-5d25-4767-952a-547657cbd6d0-kube-api-access-6c6q4" (OuterVolumeSpecName: "kube-api-access-6c6q4") pod "af3680c6-5d25-4767-952a-547657cbd6d0" (UID: "af3680c6-5d25-4767-952a-547657cbd6d0"). InnerVolumeSpecName "kube-api-access-6c6q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:34:39.511424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.511400 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af3680c6-5d25-4767-952a-547657cbd6d0" (UID: "af3680c6-5d25-4767-952a-547657cbd6d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:34:39.511944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.511919 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "af3680c6-5d25-4767-952a-547657cbd6d0" (UID: "af3680c6-5d25-4767-952a-547657cbd6d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:34:39.556849 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.556809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:39.610208 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.610175 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:34:39.610208 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.610203 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3680c6-5d25-4767-952a-547657cbd6d0-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:34:39.610208 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.610215 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6q4\" (UniqueName: \"kubernetes.io/projected/af3680c6-5d25-4767-952a-547657cbd6d0-kube-api-access-6c6q4\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:34:39.610416 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.610224 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3680c6-5d25-4767-952a-547657cbd6d0-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:34:39.629938 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.629904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:39.716995 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:39.716907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:40.049970 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.049940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:40.156561 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.156529 2579 generic.go:356] "Generic (PLEG): container finished" podID="af3680c6-5d25-4767-952a-547657cbd6d0" containerID="a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed" exitCode=0 Sep 11 18:34:40.156754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.156586 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" Sep 11 18:34:40.156754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.156618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" event={"ID":"af3680c6-5d25-4767-952a-547657cbd6d0","Type":"ContainerDied","Data":"a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed"} Sep 11 18:34:40.156754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.156696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc" event={"ID":"af3680c6-5d25-4767-952a-547657cbd6d0","Type":"ContainerDied","Data":"0361a957281d4fe3be89c0b381e964e42581c969c4ffc293b46d131945dc3916"} Sep 11 18:34:40.156754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.156731 2579 scope.go:117] "RemoveContainer" containerID="a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed" Sep 11 18:34:40.166576 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.166558 2579 scope.go:117] "RemoveContainer" containerID="a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed" Sep 11 18:34:40.166888 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:34:40.166859 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed\": container with ID starting with a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed not found: ID does not exist" containerID="a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed" Sep 11 18:34:40.166958 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.166898 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed"} err="failed to get container status \"a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed\": rpc error: code = NotFound desc = could not find container \"a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed\": container with ID starting with a0a6488db20b1f41b013544cc245c820e5fa74164c9995746a7f2114757e6eed not found: ID does not exist" Sep 11 18:34:40.181216 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.181193 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc"] Sep 11 18:34:40.189241 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.189217 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b2b4b09-54966496f9-mwdmc"] Sep 11 18:34:40.229462 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.229432 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af3680c6-5d25-4767-952a-547657cbd6d0" path="/var/lib/kubelet/pods/af3680c6-5d25-4767-952a-547657cbd6d0/volumes" Sep 11 18:34:40.390389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.390309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:40.477300 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.477251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:40.962521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:40.962482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:41.899098 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:41.899062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:41.998185 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:41.998150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:42.713974 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:42.713936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:43.518108 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:43.518075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:44.253751 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:44.253713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:47.642374 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:47.642330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:47.655187 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:47.655146 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:34:48.494885 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:48.494845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:48.610635 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:48.610558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:48.695799 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:48.695751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:48.798339 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:48.798302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:49.164198 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:49.164095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:49.734410 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:49.734374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:50.705030 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:50.704996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:51.472265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:51.472228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:52.269462 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:52.269418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:53.212495 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:53.212461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:58.024984 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:58.024890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:58.129426 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:58.129390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:58.229061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:58.229022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:58.420569 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:58.420468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:58.496411 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:58.496375 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:58.810312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:58.810277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:58.893693 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:58.893656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:59.605857 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:34:59.605816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:00.491489 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:00.491453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:01.361613 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:01.361580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:02.166220 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:02.166183 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:03.867054 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:03.867004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:07.054279 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:07.054243 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:35:14.324410 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:14.324368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:15.069263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:15.069226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:21.237840 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:21.237801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:21.384588 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:21.384552 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:21.469319 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:21.469273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:32.297730 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:32.297682 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:35:32.300101 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:32.300078 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:35:32.442600 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:32.442567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:37.320488 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:37.320438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:37.981979 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:37.981947 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:35:41.626404 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:41.626368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:41.870347 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:41.870315 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4"] Sep 11 18:35:41.870559 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:41.870535 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" containerID="cri-o://e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546" gracePeriod=30 Sep 11 18:35:41.902922 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:41.902834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:42.330651 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.330621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:42.651686 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.651597 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn"] Sep 11 18:35:42.652089 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.652031 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3680c6-5d25-4767-952a-547657cbd6d0" containerName="maas-api" Sep 11 18:35:42.652089 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.652043 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3680c6-5d25-4767-952a-547657cbd6d0" containerName="maas-api" Sep 11 18:35:42.652162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.652124 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3680c6-5d25-4767-952a-547657cbd6d0" containerName="maas-api" Sep 11 18:35:42.656566 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.656546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.659251 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.659230 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-a228c4-openshift-default-dockercfg-vb9dg\"" Sep 11 18:35:42.671197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.671169 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn"] Sep 11 18:35:42.682011 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.681979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:42.732990 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.732956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-certs\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.733173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.733008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8ln8\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-kube-api-access-c8ln8\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.733173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.733033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-credential-socket\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.733173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.733054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-socket\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.733173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.733071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-token\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.733173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.733099 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-envoy\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.733173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.733161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-podinfo\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.733424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.733289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istiod-ca-cert\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.733424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.733326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-data\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.773323 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.773288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:42.833963 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.833922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-socket\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.833963 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.833962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-token\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.833989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-envoy\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-podinfo\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istiod-ca-cert\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-data\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-certs\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8ln8\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-kube-api-access-c8ln8\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-credential-socket\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834592 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-socket\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-data\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-certs\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834681 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-credential-socket\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.834903 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.834883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istiod-ca-cert\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.836312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.836291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-envoy\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.836665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.836644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-podinfo\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.847427 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.847403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-token\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.847666 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.847648 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8ln8\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-kube-api-access-c8ln8\") pod \"e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:42.967073 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:42.966975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:43.247252 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:43.247229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn"] Sep 11 18:35:43.249363 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:35:43.249334 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb58aa2_de56_4576_a8e5_bfc76d1c0164.slice/crio-c3bc183203b750c06cb9219239cda934a703ce4d39f935290ceaef74e0abd4dd WatchSource:0}: Error finding container c3bc183203b750c06cb9219239cda934a703ce4d39f935290ceaef74e0abd4dd: Status 404 returned error can't find the container with id c3bc183203b750c06cb9219239cda934a703ce4d39f935290ceaef74e0abd4dd Sep 11 18:35:43.251565 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:43.251535 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:35:43.251626 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:43.251598 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:35:43.251662 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:43.251626 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:35:43.549786 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:43.549747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" event={"ID":"edb58aa2-de56-4576-a8e5-bfc76d1c0164","Type":"ContainerStarted","Data":"566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887"} Sep 11 18:35:43.549957 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:43.549807 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" event={"ID":"edb58aa2-de56-4576-a8e5-bfc76d1c0164","Type":"ContainerStarted","Data":"c3bc183203b750c06cb9219239cda934a703ce4d39f935290ceaef74e0abd4dd"} Sep 11 18:35:43.577359 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:43.577305 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" podStartSLOduration=1.577289413 podStartE2EDuration="1.577289413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:43.574368518 +0000 UTC m=+1211.801044895" watchObservedRunningTime="2026-09-11 18:35:43.577289413 +0000 UTC m=+1211.803965754" Sep 11 18:35:43.967158 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:43.967073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:44.064897 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:44.064862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:44.168833 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:44.168784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:44.284249 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:44.284202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:44.889066 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:44.889018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:44.972417 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:44.972386 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:45.562520 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:45.562490 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:45.563305 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:45.563285 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:35:45.670789 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:45.670756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:45.809054 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:45.809006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:45.995606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:45.995516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:46.129255 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:46.129221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:47.122567 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.122543 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:35:47.199420 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.199329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:47.278187 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278154 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-socket\") pod \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " Sep 11 18:35:47.278389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-token\") pod \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " Sep 11 18:35:47.278389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278266 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-envoy\") pod \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " Sep 11 18:35:47.278389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278310 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istiod-ca-cert\") pod \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " Sep 11 18:35:47.278389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278347 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-data\") pod \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " Sep 11 18:35:47.278621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278411 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-credential-socket\") pod \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " Sep 11 18:35:47.278621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278414 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23deb4f6-f0dd-4f24-943c-b98a903d5b39" (UID: "23deb4f6-f0dd-4f24-943c-b98a903d5b39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:47.278621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278446 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwbrt\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-kube-api-access-dwbrt\") pod \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " Sep 11 18:35:47.278621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278481 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-certs\") pod \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " Sep 11 18:35:47.278621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278520 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-podinfo\") pod \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\" (UID: \"23deb4f6-f0dd-4f24-943c-b98a903d5b39\") " Sep 11 18:35:47.278928 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278726 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-data" (OuterVolumeSpecName: "istio-data") pod "23deb4f6-f0dd-4f24-943c-b98a903d5b39" (UID: "23deb4f6-f0dd-4f24-943c-b98a903d5b39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:47.278928 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23deb4f6-f0dd-4f24-943c-b98a903d5b39" (UID: "23deb4f6-f0dd-4f24-943c-b98a903d5b39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:47.278928 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278806 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23deb4f6-f0dd-4f24-943c-b98a903d5b39" (UID: "23deb4f6-f0dd-4f24-943c-b98a903d5b39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:35:47.279085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278930 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-data\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:35:47.279085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278947 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-credential-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:35:47.279085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278957 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:35:47.279085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.278965 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istiod-ca-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:35:47.279277 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.279189 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23deb4f6-f0dd-4f24-943c-b98a903d5b39" (UID: "23deb4f6-f0dd-4f24-943c-b98a903d5b39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:47.280910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.280888 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23deb4f6-f0dd-4f24-943c-b98a903d5b39" (UID: "23deb4f6-f0dd-4f24-943c-b98a903d5b39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:35:47.281066 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.281041 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23deb4f6-f0dd-4f24-943c-b98a903d5b39" (UID: "23deb4f6-f0dd-4f24-943c-b98a903d5b39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:47.281133 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.281049 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-kube-api-access-dwbrt" (OuterVolumeSpecName: "kube-api-access-dwbrt") pod "23deb4f6-f0dd-4f24-943c-b98a903d5b39" (UID: "23deb4f6-f0dd-4f24-943c-b98a903d5b39"). InnerVolumeSpecName "kube-api-access-dwbrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:35:47.281341 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.281318 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-token" (OuterVolumeSpecName: "istio-token") pod "23deb4f6-f0dd-4f24-943c-b98a903d5b39" (UID: "23deb4f6-f0dd-4f24-943c-b98a903d5b39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:35:47.309403 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.309372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:47.379558 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.379523 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbrt\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-kube-api-access-dwbrt\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:35:47.379558 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.379553 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-workload-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:35:47.379558 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.379566 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-podinfo\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:35:47.379822 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.379574 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:35:47.379822 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.379582 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23deb4f6-f0dd-4f24-943c-b98a903d5b39-istio-envoy\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:35:47.452070 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.451969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:47.576274 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.576238 2579 generic.go:356] "Generic (PLEG): container finished" podID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerID="e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546" exitCode=0 Sep 11 18:35:47.576446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.576293 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" Sep 11 18:35:47.576446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.576335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" event={"ID":"23deb4f6-f0dd-4f24-943c-b98a903d5b39","Type":"ContainerDied","Data":"e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546"} Sep 11 18:35:47.576446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.576390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4" event={"ID":"23deb4f6-f0dd-4f24-943c-b98a903d5b39","Type":"ContainerDied","Data":"fdd43a47e4f74bba3780d8a9cae4c8cbf156794bce162e067977e99717719930"} Sep 11 18:35:47.576446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.576411 2579 scope.go:117] "RemoveContainer" containerID="e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546" Sep 11 18:35:47.588133 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.588108 2579 scope.go:117] "RemoveContainer" containerID="56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59" Sep 11 18:35:47.597524 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.597503 2579 scope.go:117] "RemoveContainer" containerID="e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546" Sep 11 18:35:47.597826 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:35:47.597807 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546\": container with ID starting with e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546 not found: ID does not exist" containerID="e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546" Sep 11 18:35:47.597886 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.597835 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546"} err="failed to get container status \"e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546\": rpc error: code = NotFound desc = could not find container \"e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546\": container with ID starting with e8684a91c376d4abe332b97c48909b195b0fc8b3828e0badc8ebdff03eb3c546 not found: ID does not exist" Sep 11 18:35:47.597886 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.597852 2579 scope.go:117] "RemoveContainer" containerID="56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59" Sep 11 18:35:47.598109 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:35:47.598091 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59\": container with ID starting with 56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59 not found: ID does not exist" containerID="56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59" Sep 11 18:35:47.598163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.598119 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59"} err="failed to get container status \"56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59\": rpc error: code = NotFound desc = could not find container \"56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59\": container with ID starting with 56d4c1402dbb638888391db5e8a79a83e6e1bce1e3202914fc991a9750264e59 not found: ID does not exist" Sep 11 18:35:47.613173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.613140 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4"] Sep 11 18:35:47.620751 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:47.620729 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-536a4d-openshift-default-c8cb9c9fb-kwld4"] Sep 11 18:35:48.229621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:48.229587 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" path="/var/lib/kubelet/pods/23deb4f6-f0dd-4f24-943c-b98a903d5b39/volumes" Sep 11 18:35:48.478528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:48.478492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:49.541765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:49.541721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:50.474590 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.474559 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v"] Sep 11 18:35:50.475026 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.475013 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" Sep 11 18:35:50.475070 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.475028 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" Sep 11 18:35:50.475070 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.475042 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" Sep 11 18:35:50.475070 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.475047 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" Sep 11 18:35:50.478722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.475814 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" Sep 11 18:35:50.478722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.475845 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23deb4f6-f0dd-4f24-943c-b98a903d5b39" containerName="istio-proxy" Sep 11 18:35:50.483072 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.482931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.503719 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.503681 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-a228c4-serving-cert\"" Sep 11 18:35:50.504698 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.504677 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v"] Sep 11 18:35:50.558977 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.558941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:50.612070 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.612035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-maas-api-tls\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.612228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.612075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649abe1-167d-47ec-a9be-b93979d5bde4-openshift-service-ca\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.612228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.612163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-metrics-certs\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.612321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.612239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2xff\" (UniqueName: \"kubernetes.io/projected/5649abe1-167d-47ec-a9be-b93979d5bde4-kube-api-access-n2xff\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.675354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.675314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:50.712856 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.712821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-metrics-certs\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.713032 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.712911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2xff\" (UniqueName: \"kubernetes.io/projected/5649abe1-167d-47ec-a9be-b93979d5bde4-kube-api-access-n2xff\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.713032 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.712932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-maas-api-tls\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.713032 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.712954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649abe1-167d-47ec-a9be-b93979d5bde4-openshift-service-ca\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.713666 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.713638 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649abe1-167d-47ec-a9be-b93979d5bde4-openshift-service-ca\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.715408 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.715382 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-metrics-certs\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.715492 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.715457 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-maas-api-tls\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.723448 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.723424 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2xff\" (UniqueName: \"kubernetes.io/projected/5649abe1-167d-47ec-a9be-b93979d5bde4-kube-api-access-n2xff\") pod \"maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:50.795132 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:50.795097 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:51.076688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:51.076616 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v"] Sep 11 18:35:51.080358 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:35:51.080328 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5649abe1_167d_47ec_a9be_b93979d5bde4.slice/crio-ccaa638eb838667a6af943be16d5d402ad21a0a9c75cbf555b91d15b7d6191af WatchSource:0}: Error finding container ccaa638eb838667a6af943be16d5d402ad21a0a9c75cbf555b91d15b7d6191af: Status 404 returned error can't find the container with id ccaa638eb838667a6af943be16d5d402ad21a0a9c75cbf555b91d15b7d6191af Sep 11 18:35:51.217482 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:51.217449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:51.322916 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:51.322877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:51.604051 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:51.604017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" event={"ID":"5649abe1-167d-47ec-a9be-b93979d5bde4","Type":"ContainerStarted","Data":"74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab"} Sep 11 18:35:51.604438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:51.604070 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" event={"ID":"5649abe1-167d-47ec-a9be-b93979d5bde4","Type":"ContainerStarted","Data":"ccaa638eb838667a6af943be16d5d402ad21a0a9c75cbf555b91d15b7d6191af"} Sep 11 18:35:51.604438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:51.604101 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:35:51.642317 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:51.642267 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" podStartSLOduration=1.291164484 podStartE2EDuration="1.642251615s" totalImagesPullingTime="351.087131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:51.639511349 +0000 UTC m=+1219.866187708" watchObservedRunningTime="2026-09-11 18:35:51.642251615 +0000 UTC m=+1219.868927954" Sep 11 18:35:54.831076 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:54.831040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:55.707317 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:55.707281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:57.614822 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:35:57.614790 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:36:01.281257 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:01.281219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:36:03.036755 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:03.036724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:03.143577 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:03.143530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:03.281168 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:03.281130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:03.459660 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:03.459577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:04.146761 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:04.146724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:04.265797 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:04.265756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:05.652758 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:05.652695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:05.943848 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:05.943758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:12.167320 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.167278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:12.262161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.262126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:12.482082 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.482054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:12.754448 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.754410 2579 generic.go:356] "Generic (PLEG): container finished" podID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerID="566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887" exitCode=0 Sep 11 18:36:12.754628 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.754485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" event={"ID":"edb58aa2-de56-4576-a8e5-bfc76d1c0164","Type":"ContainerDied","Data":"566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887"} Sep 11 18:36:12.754982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.754965 2579 scope.go:117] "RemoveContainer" containerID="566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887" Sep 11 18:36:12.756191 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.756161 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:36:12.756281 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.756236 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:36:12.756281 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.756266 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:36:12.826536 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:12.826500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:13.141186 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:13.141075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:13.763226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:13.763184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" event={"ID":"edb58aa2-de56-4576-a8e5-bfc76d1c0164","Type":"ContainerStarted","Data":"74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed"} Sep 11 18:36:13.967730 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:13.967662 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:36:13.968956 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:13.968931 2579 patch_prober.go:28] interesting pod/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:36:13.969063 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:13.968988 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:36:14.164057 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:14.163958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:14.813731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:14.812914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:14.955565 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:14.955526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:14.967763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:14.967730 2579 patch_prober.go:28] interesting pod/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:36:14.967917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:14.967788 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:36:15.562467 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:15.562431 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:36:15.612644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:15.612610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:15.781685 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:15.781650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:15.967745 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:15.967625 2579 patch_prober.go:28] interesting pod/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:36:15.967745 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:15.967690 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:36:16.556777 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:16.556746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:16.967844 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:16.967760 2579 patch_prober.go:28] interesting pod/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:36:16.967844 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:16.967827 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:36:17.619768 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:17.619727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:17.967892 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:17.967799 2579 patch_prober.go:28] interesting pod/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:36:17.968256 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:17.967888 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:36:18.971872 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:18.971839 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:36:18.972989 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:18.972965 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:36:19.812526 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:19.812490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:19.900273 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:19.900243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:20.050617 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.050583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:20.345448 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.345419 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5"] Sep 11 18:36:20.349475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.349456 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.354229 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.354213 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-51815453-openshift-default-dockercfg-mm29m\"" Sep 11 18:36:20.369343 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.369316 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5"] Sep 11 18:36:20.413768 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.413736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:20.418076 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.418040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-token\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.418231 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.418085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-envoy\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.418231 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.418149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-data\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.418231 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.418200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-socket\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.418386 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.418235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-certs\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.418386 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.418265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-podinfo\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.418386 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.418290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4422ae5a-92f1-42c6-8ec9-060713815eb2-istiod-ca-cert\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.418503 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.418386 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-credential-socket\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.418503 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.418413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbzx\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-kube-api-access-whbzx\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519165 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-data\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-socket\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-certs\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-podinfo\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4422ae5a-92f1-42c6-8ec9-060713815eb2-istiod-ca-cert\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-credential-socket\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbzx\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-kube-api-access-whbzx\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-token\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519584 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-envoy\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519837 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-data\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519837 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-socket\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.519837 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519815 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-certs\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.520008 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.519865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-credential-socket\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.520150 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.520123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4422ae5a-92f1-42c6-8ec9-060713815eb2-istiod-ca-cert\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.522096 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.522073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-podinfo\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.522216 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.522187 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-envoy\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.531841 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.531818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbzx\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-kube-api-access-whbzx\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.532124 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.532107 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-token\") pod \"e2e-mt-51815453-openshift-default-869655f45b-2fcm5\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.617095 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.617011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:20.660406 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.660373 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:20.972687 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.972660 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5"] Sep 11 18:36:20.973489 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:36:20.973467 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4422ae5a_92f1_42c6_8ec9_060713815eb2.slice/crio-ee1839404b418eeff2bb10332fcd7fac24044c9d66922cb4abac0585165132bb WatchSource:0}: Error finding container ee1839404b418eeff2bb10332fcd7fac24044c9d66922cb4abac0585165132bb: Status 404 returned error can't find the container with id ee1839404b418eeff2bb10332fcd7fac24044c9d66922cb4abac0585165132bb Sep 11 18:36:20.975697 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.975663 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:36:20.975804 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.975744 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:36:20.975804 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:20.975772 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:36:21.816377 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:21.816337 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" event={"ID":"4422ae5a-92f1-42c6-8ec9-060713815eb2","Type":"ContainerStarted","Data":"b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6"} Sep 11 18:36:21.816927 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:21.816407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" event={"ID":"4422ae5a-92f1-42c6-8ec9-060713815eb2","Type":"ContainerStarted","Data":"ee1839404b418eeff2bb10332fcd7fac24044c9d66922cb4abac0585165132bb"} Sep 11 18:36:21.841876 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:21.841791 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" podStartSLOduration=1.841769995 podStartE2EDuration="1.841769995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:21.839013704 +0000 UTC m=+1250.065690044" watchObservedRunningTime="2026-09-11 18:36:21.841769995 +0000 UTC m=+1250.068446336" Sep 11 18:36:22.661297 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:22.661253 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:22.666347 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:22.666321 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:22.823076 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:22.823043 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:22.824099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:22.824079 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:26.770767 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:26.770674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:27.455825 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:27.455788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:27.659842 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:27.659806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:28.027437 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:28.027399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:29.428819 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:29.428788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:30.603257 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:30.603225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:32.015227 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:32.015173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:33.137257 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:33.137221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:34.203672 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.203637 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749"] Sep 11 18:36:34.208234 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.208209 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.211932 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.211911 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-51815453-serving-cert\"" Sep 11 18:36:34.234571 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.234544 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749"] Sep 11 18:36:34.254824 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.254795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-metrics-certs\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.254948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.254835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfccn\" (UniqueName: \"kubernetes.io/projected/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-kube-api-access-tfccn\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.254948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.254930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-maas-api-tls\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.255022 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.254964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-openshift-service-ca\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.292061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.292030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:34.356309 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.356272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-maas-api-tls\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.356507 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.356313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-openshift-service-ca\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.356507 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.356410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-metrics-certs\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.356507 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:36:34.356415 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-51815453-serving-cert: secret "maas-api-e2e-mt-51815453-serving-cert" not found Sep 11 18:36:34.356507 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.356449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfccn\" (UniqueName: \"kubernetes.io/projected/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-kube-api-access-tfccn\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.356507 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:36:34.356479 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-maas-api-tls podName:925c8c6f-f8f7-4834-8a5e-13d5e6aad226 nodeName:}" failed. No retries permitted until 2026-09-11 18:36:34.856459737 +0000 UTC m=+1263.083136055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-maas-api-tls") pod "maas-api-e2e-mt-51815453-97c957d4d-jq749" (UID: "925c8c6f-f8f7-4834-8a5e-13d5e6aad226") : secret "maas-api-e2e-mt-51815453-serving-cert" not found Sep 11 18:36:34.356989 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.356970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-openshift-service-ca\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.358818 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.358802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-metrics-certs\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.369173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.369151 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfccn\" (UniqueName: \"kubernetes.io/projected/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-kube-api-access-tfccn\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.399125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.399094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:34.773009 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.772973 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:34.862649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.862611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-maas-api-tls\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:34.865127 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:34.865108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-maas-api-tls\") pod \"maas-api-e2e-mt-51815453-97c957d4d-jq749\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:35.114674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:35.114583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:35.118540 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:35.118518 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:35.260554 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:35.260510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:35.446203 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:36:35.446171 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925c8c6f_f8f7_4834_8a5e_13d5e6aad226.slice/crio-72e131f86657602dbd6b2e36fce51bcc4be43db78fb117ca563cad0a124d555e WatchSource:0}: Error finding container 72e131f86657602dbd6b2e36fce51bcc4be43db78fb117ca563cad0a124d555e: Status 404 returned error can't find the container with id 72e131f86657602dbd6b2e36fce51bcc4be43db78fb117ca563cad0a124d555e Sep 11 18:36:35.447292 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:35.447264 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749"] Sep 11 18:36:35.906018 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:35.905976 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" event={"ID":"925c8c6f-f8f7-4834-8a5e-13d5e6aad226","Type":"ContainerStarted","Data":"f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5"} Sep 11 18:36:35.906210 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:35.906047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" event={"ID":"925c8c6f-f8f7-4834-8a5e-13d5e6aad226","Type":"ContainerStarted","Data":"72e131f86657602dbd6b2e36fce51bcc4be43db78fb117ca563cad0a124d555e"} Sep 11 18:36:35.906210 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:35.906076 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:35.934348 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:35.934297 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" podStartSLOduration=1.5772797509999998 podStartE2EDuration="1.934280343s" totalImagesPullingTime="357.000592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:35.93209522 +0000 UTC m=+1264.158771579" watchObservedRunningTime="2026-09-11 18:36:35.934280343 +0000 UTC m=+1264.160956683" Sep 11 18:36:37.269585 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:37.269546 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:36:41.916972 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:41.916939 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:36:57.256603 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:57.256573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:57.461275 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:57.461189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:58.061568 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.061536 2579 generic.go:356] "Generic (PLEG): container finished" podID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerID="b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6" exitCode=0 Sep 11 18:36:58.061765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.061611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" event={"ID":"4422ae5a-92f1-42c6-8ec9-060713815eb2","Type":"ContainerDied","Data":"b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6"} Sep 11 18:36:58.062183 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.062161 2579 scope.go:117] "RemoveContainer" containerID="b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6" Sep 11 18:36:58.063351 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.063324 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:36:58.063438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.063417 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:36:58.063495 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.063472 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:36:58.145572 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.145529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:58.278595 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.278558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:58.399673 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.399580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:58.588259 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:58.588223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:59.069591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:59.069552 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" event={"ID":"4422ae5a-92f1-42c6-8ec9-060713815eb2","Type":"ContainerStarted","Data":"aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33"} Sep 11 18:36:59.271289 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:59.271255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:59.661025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:59.660988 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:36:59.661483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:59.661266 2579 patch_prober.go:28] interesting pod/e2e-mt-51815453-openshift-default-869655f45b-2fcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:36:59.661483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:36:59.661329 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:37:00.078855 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:00.078817 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:37:00.161897 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:00.161857 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:00.661150 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:00.661114 2579 patch_prober.go:28] interesting pod/e2e-mt-51815453-openshift-default-869655f45b-2fcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:37:00.661519 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:00.661169 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:37:01.053371 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:01.053338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:01.661637 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:01.661602 2579 patch_prober.go:28] interesting pod/e2e-mt-51815453-openshift-default-869655f45b-2fcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:37:01.662033 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:01.661653 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:37:01.965099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:01.964993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:02.661424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:02.661385 2579 patch_prober.go:28] interesting pod/e2e-mt-51815453-openshift-default-869655f45b-2fcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:37:02.661603 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:02.661442 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:37:02.761192 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:02.761156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:03.660881 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:03.660843 2579 patch_prober.go:28] interesting pod/e2e-mt-51815453-openshift-default-869655f45b-2fcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:37:03.661057 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:03.660899 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:37:04.608920 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:04.608886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:04.661027 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:04.660986 2579 patch_prober.go:28] interesting pod/e2e-mt-51815453-openshift-default-869655f45b-2fcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:37:04.661203 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:04.661045 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:37:04.796619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:04.796579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:05.665200 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:05.665169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:37:05.665665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:05.665653 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:37:05.666353 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:05.666331 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:37:05.734993 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:05.734950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:06.741192 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:06.741156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:07.558098 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:07.558064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:08.539973 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:08.539931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:09.303556 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:09.303525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:10.226622 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:10.226527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:11.049656 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:11.049617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:15.288938 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.288901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:15.589506 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.589414 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749"] Sep 11 18:37:15.589877 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.589789 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" podUID="925c8c6f-f8f7-4834-8a5e-13d5e6aad226" containerName="maas-api" containerID="cri-o://f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5" gracePeriod=30 Sep 11 18:37:15.623126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.623073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:15.723047 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.722960 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:15.825377 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.825342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:15.852252 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.852191 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:37:15.893530 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.893494 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-metrics-certs\") pod \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " Sep 11 18:37:15.893740 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.893560 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-maas-api-tls\") pod \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " Sep 11 18:37:15.893740 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.893596 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfccn\" (UniqueName: \"kubernetes.io/projected/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-kube-api-access-tfccn\") pod \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " Sep 11 18:37:15.893902 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.893757 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-openshift-service-ca\") pod \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\" (UID: \"925c8c6f-f8f7-4834-8a5e-13d5e6aad226\") " Sep 11 18:37:15.894144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.894112 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "925c8c6f-f8f7-4834-8a5e-13d5e6aad226" (UID: "925c8c6f-f8f7-4834-8a5e-13d5e6aad226"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:37:15.894296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.894277 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:15.895857 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.895827 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-kube-api-access-tfccn" (OuterVolumeSpecName: "kube-api-access-tfccn") pod "925c8c6f-f8f7-4834-8a5e-13d5e6aad226" (UID: "925c8c6f-f8f7-4834-8a5e-13d5e6aad226"). InnerVolumeSpecName "kube-api-access-tfccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:37:15.895948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.895851 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "925c8c6f-f8f7-4834-8a5e-13d5e6aad226" (UID: "925c8c6f-f8f7-4834-8a5e-13d5e6aad226"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:37:15.895948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.895878 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "925c8c6f-f8f7-4834-8a5e-13d5e6aad226" (UID: "925c8c6f-f8f7-4834-8a5e-13d5e6aad226"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:37:15.923817 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.923782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:15.995602 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.995567 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:15.995602 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.995603 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:15.995836 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:15.995618 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfccn\" (UniqueName: \"kubernetes.io/projected/925c8c6f-f8f7-4834-8a5e-13d5e6aad226-kube-api-access-tfccn\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:16.198356 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.198267 2579 generic.go:356] "Generic (PLEG): container finished" podID="925c8c6f-f8f7-4834-8a5e-13d5e6aad226" containerID="f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5" exitCode=0 Sep 11 18:37:16.198356 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.198313 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" Sep 11 18:37:16.198550 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.198359 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" event={"ID":"925c8c6f-f8f7-4834-8a5e-13d5e6aad226","Type":"ContainerDied","Data":"f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5"} Sep 11 18:37:16.198550 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.198413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749" event={"ID":"925c8c6f-f8f7-4834-8a5e-13d5e6aad226","Type":"ContainerDied","Data":"72e131f86657602dbd6b2e36fce51bcc4be43db78fb117ca563cad0a124d555e"} Sep 11 18:37:16.198550 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.198438 2579 scope.go:117] "RemoveContainer" containerID="f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5" Sep 11 18:37:16.208085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.208070 2579 scope.go:117] "RemoveContainer" containerID="f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5" Sep 11 18:37:16.208360 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:37:16.208339 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5\": container with ID starting with f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5 not found: ID does not exist" containerID="f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5" Sep 11 18:37:16.208430 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.208366 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5"} err="failed to get container status \"f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5\": rpc error: code = NotFound desc = could not find container \"f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5\": container with ID starting with f877c9d997658f9ce2a68ca71bd9732f67fcb0768eb72787f06ee8fcebd66dd5 not found: ID does not exist" Sep 11 18:37:16.226520 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.226440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:16.231273 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.231251 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749"] Sep 11 18:37:16.231387 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.231282 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51815453-97c957d4d-jq749"] Sep 11 18:37:16.324348 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:16.324316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:17.347366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:17.347332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:18.229508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:18.229475 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="925c8c6f-f8f7-4834-8a5e-13d5e6aad226" path="/var/lib/kubelet/pods/925c8c6f-f8f7-4834-8a5e-13d5e6aad226/volumes" Sep 11 18:37:18.359271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:18.359238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:19.159043 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:19.158998 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:20.127490 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:20.127446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:21.028120 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:21.028080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:22.996376 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:22.996340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:23.096914 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:23.096878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:23.220756 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:23.220723 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5"] Sep 11 18:37:23.220977 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:23.220951 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" containerID="cri-o://aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33" gracePeriod=30 Sep 11 18:37:23.244549 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:23.244493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:23.627443 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:23.627398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:23.732907 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:23.732869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:23.835444 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:23.835402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:25.828278 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:25.828238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:26.636659 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:26.636615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:28.123834 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.123797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:28.473689 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.473661 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:37:28.622971 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.622936 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-socket\") pod \"4422ae5a-92f1-42c6-8ec9-060713815eb2\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " Sep 11 18:37:28.623163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.622989 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-certs\") pod \"4422ae5a-92f1-42c6-8ec9-060713815eb2\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " Sep 11 18:37:28.623163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623026 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-token\") pod \"4422ae5a-92f1-42c6-8ec9-060713815eb2\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " Sep 11 18:37:28.623163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623094 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-data\") pod \"4422ae5a-92f1-42c6-8ec9-060713815eb2\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " Sep 11 18:37:28.623402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623209 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4422ae5a-92f1-42c6-8ec9-060713815eb2-istiod-ca-cert\") pod \"4422ae5a-92f1-42c6-8ec9-060713815eb2\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " Sep 11 18:37:28.623402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623272 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whbzx\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-kube-api-access-whbzx\") pod \"4422ae5a-92f1-42c6-8ec9-060713815eb2\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " Sep 11 18:37:28.623402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623274 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4422ae5a-92f1-42c6-8ec9-060713815eb2" (UID: "4422ae5a-92f1-42c6-8ec9-060713815eb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:37:28.623402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623306 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-podinfo\") pod \"4422ae5a-92f1-42c6-8ec9-060713815eb2\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " Sep 11 18:37:28.623402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623333 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4422ae5a-92f1-42c6-8ec9-060713815eb2" (UID: "4422ae5a-92f1-42c6-8ec9-060713815eb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:37:28.623402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623344 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-data" (OuterVolumeSpecName: "istio-data") pod "4422ae5a-92f1-42c6-8ec9-060713815eb2" (UID: "4422ae5a-92f1-42c6-8ec9-060713815eb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:37:28.623402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623396 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-envoy\") pod \"4422ae5a-92f1-42c6-8ec9-060713815eb2\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " Sep 11 18:37:28.623763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623442 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-credential-socket\") pod \"4422ae5a-92f1-42c6-8ec9-060713815eb2\" (UID: \"4422ae5a-92f1-42c6-8ec9-060713815eb2\") " Sep 11 18:37:28.623763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623557 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4422ae5a-92f1-42c6-8ec9-060713815eb2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4422ae5a-92f1-42c6-8ec9-060713815eb2" (UID: "4422ae5a-92f1-42c6-8ec9-060713815eb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:37:28.623763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623672 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4422ae5a-92f1-42c6-8ec9-060713815eb2" (UID: "4422ae5a-92f1-42c6-8ec9-060713815eb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:37:28.623974 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623953 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4422ae5a-92f1-42c6-8ec9-060713815eb2-istiod-ca-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:28.624059 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623981 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-credential-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:28.624059 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.623995 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:28.624059 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.624009 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-workload-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:28.624059 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.624022 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-data\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:28.625767 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.625742 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4422ae5a-92f1-42c6-8ec9-060713815eb2" (UID: "4422ae5a-92f1-42c6-8ec9-060713815eb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:37:28.625899 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.625809 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-kube-api-access-whbzx" (OuterVolumeSpecName: "kube-api-access-whbzx") pod "4422ae5a-92f1-42c6-8ec9-060713815eb2" (UID: "4422ae5a-92f1-42c6-8ec9-060713815eb2"). InnerVolumeSpecName "kube-api-access-whbzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:37:28.625899 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.625872 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-token" (OuterVolumeSpecName: "istio-token") pod "4422ae5a-92f1-42c6-8ec9-060713815eb2" (UID: "4422ae5a-92f1-42c6-8ec9-060713815eb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:37:28.626100 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.626075 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4422ae5a-92f1-42c6-8ec9-060713815eb2" (UID: "4422ae5a-92f1-42c6-8ec9-060713815eb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:37:28.724652 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.724560 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:28.724652 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.724592 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whbzx\" (UniqueName: \"kubernetes.io/projected/4422ae5a-92f1-42c6-8ec9-060713815eb2-kube-api-access-whbzx\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:28.724652 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.724602 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-podinfo\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:28.724652 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:28.724611 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4422ae5a-92f1-42c6-8ec9-060713815eb2-istio-envoy\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:29.285495 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.285463 2579 generic.go:356] "Generic (PLEG): container finished" podID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerID="aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33" exitCode=0 Sep 11 18:37:29.285951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.285520 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" Sep 11 18:37:29.285951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.285548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" event={"ID":"4422ae5a-92f1-42c6-8ec9-060713815eb2","Type":"ContainerDied","Data":"aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33"} Sep 11 18:37:29.285951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.285604 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5" event={"ID":"4422ae5a-92f1-42c6-8ec9-060713815eb2","Type":"ContainerDied","Data":"ee1839404b418eeff2bb10332fcd7fac24044c9d66922cb4abac0585165132bb"} Sep 11 18:37:29.285951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.285634 2579 scope.go:117] "RemoveContainer" containerID="aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33" Sep 11 18:37:29.297232 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.297212 2579 scope.go:117] "RemoveContainer" containerID="b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6" Sep 11 18:37:29.305992 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.305968 2579 scope.go:117] "RemoveContainer" containerID="aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33" Sep 11 18:37:29.306263 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:37:29.306247 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33\": container with ID starting with aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33 not found: ID does not exist" containerID="aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33" Sep 11 18:37:29.306306 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.306271 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33"} err="failed to get container status \"aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33\": rpc error: code = NotFound desc = could not find container \"aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33\": container with ID starting with aa45a858dbdb679215066047b33c36c171209f5e8d406dbcfc1bad9b6ba5db33 not found: ID does not exist" Sep 11 18:37:29.306306 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.306288 2579 scope.go:117] "RemoveContainer" containerID="b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6" Sep 11 18:37:29.306559 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:37:29.306538 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6\": container with ID starting with b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6 not found: ID does not exist" containerID="b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6" Sep 11 18:37:29.306645 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.306565 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6"} err="failed to get container status \"b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6\": rpc error: code = NotFound desc = could not find container \"b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6\": container with ID starting with b7012bbf9291a5e7d65086d24408b355da086eb47ac842b83d11c760b958e6f6 not found: ID does not exist" Sep 11 18:37:29.311124 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.311103 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5"] Sep 11 18:37:29.314205 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.314185 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-51815453-openshift-default-869655f45b-2fcm5"] Sep 11 18:37:29.905508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:29.905475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:30.075172 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:30.075136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:30.166893 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:30.166804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:30.230068 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:30.230036 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" path="/var/lib/kubelet/pods/4422ae5a-92f1-42c6-8ec9-060713815eb2/volumes" Sep 11 18:37:30.279287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:30.279256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:32.159541 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:32.159507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:32.868251 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:32.868212 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:37:36.475636 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:36.475598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:36.748258 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:36.748168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:37.072728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:37.072664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:37.181538 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:37.181500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:37.838233 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:37.838194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:37.958863 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:37.958825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:38.355882 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.355846 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q"] Sep 11 18:37:38.356304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.356290 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" Sep 11 18:37:38.356304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.356305 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" Sep 11 18:37:38.356400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.356314 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925c8c6f-f8f7-4834-8a5e-13d5e6aad226" containerName="maas-api" Sep 11 18:37:38.356400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.356322 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="925c8c6f-f8f7-4834-8a5e-13d5e6aad226" containerName="maas-api" Sep 11 18:37:38.356400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.356338 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" Sep 11 18:37:38.356400 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.356344 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" Sep 11 18:37:38.356525 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.356423 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" Sep 11 18:37:38.356525 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.356434 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="925c8c6f-f8f7-4834-8a5e-13d5e6aad226" containerName="maas-api" Sep 11 18:37:38.361013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.360993 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.365826 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.365806 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-72c98d-openshift-default-dockercfg-2x8mb\"" Sep 11 18:37:38.378611 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.378588 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q"] Sep 11 18:37:38.401623 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.401575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:38.518281 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.518245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-credential-socket\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.518463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.518302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-certs\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.518463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.518320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-token\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.518463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.518368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-podinfo\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.518463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.518390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-envoy\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.518463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.518405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-data\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.518463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.518423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-socket\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.518463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.518452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istiod-ca-cert\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.518743 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.518476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblkr\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-kube-api-access-hblkr\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619201 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-certs\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619201 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-token\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619201 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-podinfo\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619496 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-envoy\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619496 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-data\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619496 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-socket\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619655 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istiod-ca-cert\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619655 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblkr\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-kube-api-access-hblkr\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619799 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-data\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619799 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-socket\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619799 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-credential-socket\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.619996 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.619976 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-certs\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.620089 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.620067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-credential-socket\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.620380 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.620358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istiod-ca-cert\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.622179 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.622152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-podinfo\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.622294 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.622247 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-envoy\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.628028 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.628008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblkr\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-kube-api-access-hblkr\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.630548 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.630528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-token\") pod \"e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.671973 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.671944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:38.951748 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.951630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:38.959114 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:37:38.959085 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492f29e9_c3aa_413d_9e32_8e7fd0c1c1e7.slice/crio-de42f52acdd2dc080423b8c55962865cba419c367ca20d8d56a706d5cbd0add4 WatchSource:0}: Error finding container de42f52acdd2dc080423b8c55962865cba419c367ca20d8d56a706d5cbd0add4: Status 404 returned error can't find the container with id de42f52acdd2dc080423b8c55962865cba419c367ca20d8d56a706d5cbd0add4 Sep 11 18:37:38.959281 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.959250 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q"] Sep 11 18:37:38.961664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.961621 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:37:38.961817 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.961742 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:37:38.961872 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:38.961813 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:37:39.353597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:39.353562 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" event={"ID":"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7","Type":"ContainerStarted","Data":"1c78486fd2c4d589155745c8328c28833433146edeb95d84ac5b1278ea1dffd8"} Sep 11 18:37:39.353793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:39.353620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" event={"ID":"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7","Type":"ContainerStarted","Data":"de42f52acdd2dc080423b8c55962865cba419c367ca20d8d56a706d5cbd0add4"} Sep 11 18:37:39.378008 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:39.377965 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" podStartSLOduration=1.377949794 podStartE2EDuration="1.377949794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:39.376931928 +0000 UTC m=+1327.603608288" watchObservedRunningTime="2026-09-11 18:37:39.377949794 +0000 UTC m=+1327.604626135" Sep 11 18:37:39.384654 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:39.384626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:39.672551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:39.672463 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:39.774402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:39.774360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:40.677458 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:40.677424 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:40.769126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:40.769093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:41.308108 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:41.308076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:41.365766 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:41.365740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:41.366627 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:41.366609 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:37:41.399189 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:41.399158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:41.745955 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:41.745844 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:42.829319 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:42.829282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:43.700275 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.700244 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z"] Sep 11 18:37:43.700982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.700958 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4422ae5a-92f1-42c6-8ec9-060713815eb2" containerName="istio-proxy" Sep 11 18:37:43.704677 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.704657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.707233 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.707212 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-72c98d-serving-cert\"" Sep 11 18:37:43.716194 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.716168 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z"] Sep 11 18:37:43.741620 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.741578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:43.763612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.763572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.763612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.763608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.763830 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.763626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tg96\" (UniqueName: \"kubernetes.io/projected/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-kube-api-access-2tg96\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.763830 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.763747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.843164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.843119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:43.864283 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.864250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.864455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.864290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.864455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.864309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tg96\" (UniqueName: \"kubernetes.io/projected/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-kube-api-access-2tg96\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.864455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.864378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.864637 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:37:43.864525 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-serving-cert: secret "maas-api-e2e-shared-a-w2-72c98d-serving-cert" not found Sep 11 18:37:43.864637 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:37:43.864619 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-maas-api-tls podName:d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a nodeName:}" failed. No retries permitted until 2026-09-11 18:37:44.364597604 +0000 UTC m=+1332.591273935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-maas-api-tls") pod "maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" (UID: "d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a") : secret "maas-api-e2e-shared-a-w2-72c98d-serving-cert" not found Sep 11 18:37:43.865041 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.865020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.866738 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.866698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.873660 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.873637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tg96\" (UniqueName: \"kubernetes.io/projected/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-kube-api-access-2tg96\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:43.993282 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:43.993199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:44.263927 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:44.263900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:44.357383 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:44.357347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:44.368917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:44.368886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:44.371247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:44.371218 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:44.617803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:44.617689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:44.691331 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:44.691253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:44.907945 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:44.907918 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z"] Sep 11 18:37:44.909752 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:37:44.909678 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99ec185_85dd_4ea7_9a4b_eb3cb2c2e84a.slice/crio-335745e38784ec41c54735396023a52ddacd785ce86ceb8c8cc4f5164dae0112 WatchSource:0}: Error finding container 335745e38784ec41c54735396023a52ddacd785ce86ceb8c8cc4f5164dae0112: Status 404 returned error can't find the container with id 335745e38784ec41c54735396023a52ddacd785ce86ceb8c8cc4f5164dae0112 Sep 11 18:37:44.911038 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:44.911014 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:37:45.392282 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:45.392242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" event={"ID":"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a","Type":"ContainerStarted","Data":"5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440"} Sep 11 18:37:45.392436 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:45.392296 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" event={"ID":"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a","Type":"ContainerStarted","Data":"335745e38784ec41c54735396023a52ddacd785ce86ceb8c8cc4f5164dae0112"} Sep 11 18:37:45.392436 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:45.392319 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:45.412996 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:45.412940 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" podStartSLOduration=2.053682099 podStartE2EDuration="2.41292382s" totalImagesPullingTime="359.241721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:45.410327513 +0000 UTC m=+1333.637003889" watchObservedRunningTime="2026-09-11 18:37:45.41292382 +0000 UTC m=+1333.639600160" Sep 11 18:37:45.557556 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:45.557470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:46.292917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:46.292880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:47.118634 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:47.118595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:47.901580 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:47.901542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:48.512105 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.512072 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v"] Sep 11 18:37:48.512337 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.512312 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" podUID="5649abe1-167d-47ec-a9be-b93979d5bde4" containerName="maas-api" containerID="cri-o://74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab" gracePeriod=30 Sep 11 18:37:48.540732 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.540666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:48.643731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.643685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:48.744004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.743956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:48.777248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.777190 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:37:48.802829 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.802796 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-metrics-certs\") pod \"5649abe1-167d-47ec-a9be-b93979d5bde4\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " Sep 11 18:37:48.803013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.802838 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2xff\" (UniqueName: \"kubernetes.io/projected/5649abe1-167d-47ec-a9be-b93979d5bde4-kube-api-access-n2xff\") pod \"5649abe1-167d-47ec-a9be-b93979d5bde4\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " Sep 11 18:37:48.803013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.802902 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-maas-api-tls\") pod \"5649abe1-167d-47ec-a9be-b93979d5bde4\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " Sep 11 18:37:48.803013 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.803011 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649abe1-167d-47ec-a9be-b93979d5bde4-openshift-service-ca\") pod \"5649abe1-167d-47ec-a9be-b93979d5bde4\" (UID: \"5649abe1-167d-47ec-a9be-b93979d5bde4\") " Sep 11 18:37:48.803594 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.803555 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5649abe1-167d-47ec-a9be-b93979d5bde4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5649abe1-167d-47ec-a9be-b93979d5bde4" (UID: "5649abe1-167d-47ec-a9be-b93979d5bde4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:37:48.805313 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.805281 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5649abe1-167d-47ec-a9be-b93979d5bde4" (UID: "5649abe1-167d-47ec-a9be-b93979d5bde4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:37:48.805421 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.805398 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5649abe1-167d-47ec-a9be-b93979d5bde4-kube-api-access-n2xff" (OuterVolumeSpecName: "kube-api-access-n2xff") pod "5649abe1-167d-47ec-a9be-b93979d5bde4" (UID: "5649abe1-167d-47ec-a9be-b93979d5bde4"). InnerVolumeSpecName "kube-api-access-n2xff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:37:48.805489 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.805448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5649abe1-167d-47ec-a9be-b93979d5bde4" (UID: "5649abe1-167d-47ec-a9be-b93979d5bde4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:37:48.841647 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.841617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:48.903831 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.903796 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649abe1-167d-47ec-a9be-b93979d5bde4-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:48.903831 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.903824 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:48.903831 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.903834 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2xff\" (UniqueName: \"kubernetes.io/projected/5649abe1-167d-47ec-a9be-b93979d5bde4-kube-api-access-n2xff\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:48.903831 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:48.903842 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649abe1-167d-47ec-a9be-b93979d5bde4-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:37:49.184779 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.184666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:49.290146 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.290112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:49.425133 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.425097 2579 generic.go:356] "Generic (PLEG): container finished" podID="5649abe1-167d-47ec-a9be-b93979d5bde4" containerID="74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab" exitCode=0 Sep 11 18:37:49.425309 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.425149 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" Sep 11 18:37:49.425309 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.425178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" event={"ID":"5649abe1-167d-47ec-a9be-b93979d5bde4","Type":"ContainerDied","Data":"74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab"} Sep 11 18:37:49.425309 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.425230 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v" event={"ID":"5649abe1-167d-47ec-a9be-b93979d5bde4","Type":"ContainerDied","Data":"ccaa638eb838667a6af943be16d5d402ad21a0a9c75cbf555b91d15b7d6191af"} Sep 11 18:37:49.425309 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.425253 2579 scope.go:117] "RemoveContainer" containerID="74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab" Sep 11 18:37:49.435489 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.435472 2579 scope.go:117] "RemoveContainer" containerID="74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab" Sep 11 18:37:49.435775 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:37:49.435756 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab\": container with ID starting with 74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab not found: ID does not exist" containerID="74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab" Sep 11 18:37:49.435824 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.435784 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab"} err="failed to get container status \"74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab\": rpc error: code = NotFound desc = could not find container \"74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab\": container with ID starting with 74e3aedb49f556ce39ad86753d68e592953b11742f7e44f7603ea0a9550edfab not found: ID does not exist" Sep 11 18:37:49.449954 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.449931 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v"] Sep 11 18:37:49.456759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.456739 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a228c4-768cf499ff-f9j7v"] Sep 11 18:37:49.990176 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:49.990133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:50.229757 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:50.229699 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5649abe1-167d-47ec-a9be-b93979d5bde4" path="/var/lib/kubelet/pods/5649abe1-167d-47ec-a9be-b93979d5bde4/volumes" Sep 11 18:37:51.000025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:50.999987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:51.412943 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:51.412914 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:37:52.028439 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:52.028405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:53.011752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:53.011688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:53.907450 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:53.907406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:54.639872 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:54.639769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:55.427145 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:55.427108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:55.617624 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:55.617591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:55.794853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:55.794810 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn"] Sep 11 18:37:55.795156 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:55.795063 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" containerID="cri-o://74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed" gracePeriod=30 Sep 11 18:37:55.798888 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:55.798860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:55.917263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:55.917228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:56.017059 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.017023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:56.204423 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.204331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:56.340913 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.340867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:56.538468 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.538428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:56.625442 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.625399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:56.719666 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.719620 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76"] Sep 11 18:37:56.720266 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.720174 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5649abe1-167d-47ec-a9be-b93979d5bde4" containerName="maas-api" Sep 11 18:37:56.720266 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.720193 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5649abe1-167d-47ec-a9be-b93979d5bde4" containerName="maas-api" Sep 11 18:37:56.720445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.720311 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5649abe1-167d-47ec-a9be-b93979d5bde4" containerName="maas-api" Sep 11 18:37:56.725190 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.725168 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.729824 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.729805 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-238bae-openshift-default-dockercfg-kdbcd\"" Sep 11 18:37:56.730905 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.730789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:56.735446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.735419 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76"] Sep 11 18:37:56.823817 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.823722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzkw\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-kube-api-access-zmzkw\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.824043 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.824019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-data\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.824130 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.824077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-envoy\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.824189 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.824156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-socket\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.824249 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.824205 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-certs\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.824303 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.824247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c95730-6b8e-47f5-8eac-c8c971da7722-istiod-ca-cert\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.824303 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.824276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-podinfo\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.824417 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.824323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-token\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.824417 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.824363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-credential-socket\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.830917 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.830880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:56.924531 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.924495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:56.924856 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.924834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-envoy\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.924926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.924881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-socket\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.924926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.924914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-certs\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925036 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.924940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c95730-6b8e-47f5-8eac-c8c971da7722-istiod-ca-cert\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925036 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.924966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-podinfo\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925036 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.925006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-token\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925183 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.925042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-credential-socket\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925183 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.925119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzkw\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-kube-api-access-zmzkw\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925264 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.925191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-data\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925552 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.925528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-certs\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.925646 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-data\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.925685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c95730-6b8e-47f5-8eac-c8c971da7722-istiod-ca-cert\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.925693 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-credential-socket\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.925948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.925804 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-socket\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.927636 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.927614 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-envoy\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.927872 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.927852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-podinfo\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.933685 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.933665 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-token\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:56.934212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:56.934192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzkw\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-kube-api-access-zmzkw\") pod \"e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:57.037981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:57.037938 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:57.349563 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:57.349532 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76"] Sep 11 18:37:57.351007 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:37:57.350978 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c95730_6b8e_47f5_8eac_c8c971da7722.slice/crio-ff4dde36e0312a2cb8ed06d1e2be292eea3ce3471fa745d4b67a158ed1512a7f WatchSource:0}: Error finding container ff4dde36e0312a2cb8ed06d1e2be292eea3ce3471fa745d4b67a158ed1512a7f: Status 404 returned error can't find the container with id ff4dde36e0312a2cb8ed06d1e2be292eea3ce3471fa745d4b67a158ed1512a7f Sep 11 18:37:57.353671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:57.353637 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:37:57.353760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:57.353718 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:37:57.353760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:57.353746 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:37:57.411604 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:57.411499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:57.476672 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:57.476638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" event={"ID":"c6c95730-6b8e-47f5-8eac-c8c971da7722","Type":"ContainerStarted","Data":"0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a"} Sep 11 18:37:57.476860 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:57.476697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" event={"ID":"c6c95730-6b8e-47f5-8eac-c8c971da7722","Type":"ContainerStarted","Data":"ff4dde36e0312a2cb8ed06d1e2be292eea3ce3471fa745d4b67a158ed1512a7f"} Sep 11 18:37:57.503458 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:57.503393 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" podStartSLOduration=1.5033706279999999 podStartE2EDuration="1.503370628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:57.500876084 +0000 UTC m=+1345.727552449" watchObservedRunningTime="2026-09-11 18:37:57.503370628 +0000 UTC m=+1345.730046969" Sep 11 18:37:58.038994 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:58.038946 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:58.040892 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:58.040862 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 18:37:58.041046 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:58.040914 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 18:37:58.261234 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:58.261193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:58.374749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:58.374642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:58.469930 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:58.469901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:58.570498 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:58.570468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:59.043011 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:59.042979 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:59.440268 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:59.440215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:59.494270 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:59.494236 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:59.495326 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:59.495301 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:37:59.546405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:59.546363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:37:59.632643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:37:59.632607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:00.613210 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:00.613161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:01.076196 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.076167 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:38:01.171131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171048 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8ln8\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-kube-api-access-c8ln8\") pod \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " Sep 11 18:38:01.171131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171105 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istiod-ca-cert\") pod \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " Sep 11 18:38:01.171354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171138 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-credential-socket\") pod \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " Sep 11 18:38:01.171354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171170 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-certs\") pod \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " Sep 11 18:38:01.171354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171199 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-podinfo\") pod \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " Sep 11 18:38:01.171354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171265 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-token\") pod \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " Sep 11 18:38:01.171354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171294 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-socket\") pod \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " Sep 11 18:38:01.171354 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171328 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-envoy\") pod \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " Sep 11 18:38:01.171674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171363 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-data\") pod \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\" (UID: \"edb58aa2-de56-4576-a8e5-bfc76d1c0164\") " Sep 11 18:38:01.171674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171428 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "edb58aa2-de56-4576-a8e5-bfc76d1c0164" (UID: "edb58aa2-de56-4576-a8e5-bfc76d1c0164"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:01.171674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "edb58aa2-de56-4576-a8e5-bfc76d1c0164" (UID: "edb58aa2-de56-4576-a8e5-bfc76d1c0164"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:01.171674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171611 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "edb58aa2-de56-4576-a8e5-bfc76d1c0164" (UID: "edb58aa2-de56-4576-a8e5-bfc76d1c0164"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:01.171674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171626 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "edb58aa2-de56-4576-a8e5-bfc76d1c0164" (UID: "edb58aa2-de56-4576-a8e5-bfc76d1c0164"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:38:01.171934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171746 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:01.171934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171766 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istiod-ca-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:01.171934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171781 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-credential-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:01.171934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171794 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-workload-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:01.171934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.171790 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-data" (OuterVolumeSpecName: "istio-data") pod "edb58aa2-de56-4576-a8e5-bfc76d1c0164" (UID: "edb58aa2-de56-4576-a8e5-bfc76d1c0164"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:01.173419 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.173388 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-kube-api-access-c8ln8" (OuterVolumeSpecName: "kube-api-access-c8ln8") pod "edb58aa2-de56-4576-a8e5-bfc76d1c0164" (UID: "edb58aa2-de56-4576-a8e5-bfc76d1c0164"). InnerVolumeSpecName "kube-api-access-c8ln8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:38:01.173838 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.173814 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "edb58aa2-de56-4576-a8e5-bfc76d1c0164" (UID: "edb58aa2-de56-4576-a8e5-bfc76d1c0164"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:01.174035 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.174010 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "edb58aa2-de56-4576-a8e5-bfc76d1c0164" (UID: "edb58aa2-de56-4576-a8e5-bfc76d1c0164"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:38:01.174112 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.174017 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-token" (OuterVolumeSpecName: "istio-token") pod "edb58aa2-de56-4576-a8e5-bfc76d1c0164" (UID: "edb58aa2-de56-4576-a8e5-bfc76d1c0164"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:38:01.273079 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.273036 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-envoy\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:01.273079 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.273070 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-data\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:01.273079 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.273083 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8ln8\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-kube-api-access-c8ln8\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:01.273310 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.273097 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-podinfo\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:01.273310 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.273109 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb58aa2-de56-4576-a8e5-bfc76d1c0164-istio-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:01.509982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.509950 2579 generic.go:356] "Generic (PLEG): container finished" podID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerID="74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed" exitCode=0 Sep 11 18:38:01.510139 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.510018 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" Sep 11 18:38:01.510139 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.510033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" event={"ID":"edb58aa2-de56-4576-a8e5-bfc76d1c0164","Type":"ContainerDied","Data":"74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed"} Sep 11 18:38:01.510139 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.510092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn" event={"ID":"edb58aa2-de56-4576-a8e5-bfc76d1c0164","Type":"ContainerDied","Data":"c3bc183203b750c06cb9219239cda934a703ce4d39f935290ceaef74e0abd4dd"} Sep 11 18:38:01.510139 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.510117 2579 scope.go:117] "RemoveContainer" containerID="74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed" Sep 11 18:38:01.519776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.519756 2579 scope.go:117] "RemoveContainer" containerID="566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887" Sep 11 18:38:01.530266 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.530244 2579 scope.go:117] "RemoveContainer" containerID="74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed" Sep 11 18:38:01.531604 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:01.531575 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed\": container with ID starting with 74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed not found: ID does not exist" containerID="74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed" Sep 11 18:38:01.531825 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.531617 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed"} err="failed to get container status \"74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed\": rpc error: code = NotFound desc = could not find container \"74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed\": container with ID starting with 74e950efb03d373bff073c7350fabe886d4261351285516006ce2b1df15ae0ed not found: ID does not exist" Sep 11 18:38:01.531825 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.531642 2579 scope.go:117] "RemoveContainer" containerID="566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887" Sep 11 18:38:01.532044 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:01.532021 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887\": container with ID starting with 566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887 not found: ID does not exist" containerID="566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887" Sep 11 18:38:01.532110 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.532053 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887"} err="failed to get container status \"566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887\": rpc error: code = NotFound desc = could not find container \"566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887\": container with ID starting with 566447520bab58121299d3bc5e7202afc2d83c3459faeadec96c2078ee7ea887 not found: ID does not exist" Sep 11 18:38:01.538790 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.538769 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn"] Sep 11 18:38:01.541741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.541714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:01.544392 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:01.544369 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-a228c4-openshift-default-67f4b4d7d4-x22qn"] Sep 11 18:38:02.229540 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:02.229507 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" path="/var/lib/kubelet/pods/edb58aa2-de56-4576-a8e5-bfc76d1c0164/volumes" Sep 11 18:38:02.317657 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:02.317626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:03.279303 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:03.279268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:03.527770 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:03.527740 2579 generic.go:356] "Generic (PLEG): container finished" podID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerID="1c78486fd2c4d589155745c8328c28833433146edeb95d84ac5b1278ea1dffd8" exitCode=0 Sep 11 18:38:03.527944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:03.527812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" event={"ID":"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7","Type":"ContainerDied","Data":"1c78486fd2c4d589155745c8328c28833433146edeb95d84ac5b1278ea1dffd8"} Sep 11 18:38:03.528249 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:03.528232 2579 scope.go:117] "RemoveContainer" containerID="1c78486fd2c4d589155745c8328c28833433146edeb95d84ac5b1278ea1dffd8" Sep 11 18:38:03.529423 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:03.529369 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:38:03.529483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:03.529444 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:38:03.529483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:03.529474 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:38:04.244015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:04.243984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:04.536302 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:04.536262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" event={"ID":"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7","Type":"ContainerStarted","Data":"3706159a9fdd3f1b2229daab403351e7028e483d31260769505e805fbb2beb5c"} Sep 11 18:38:04.672820 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:04.672791 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:38:04.673015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:04.672989 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 11 18:38:04.673263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:04.673226 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 11 18:38:05.032447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.032418 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z"] Sep 11 18:38:05.033126 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.033108 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" Sep 11 18:38:05.033226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.033160 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" Sep 11 18:38:05.033226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.033183 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" Sep 11 18:38:05.033226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.033192 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" Sep 11 18:38:05.033394 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.033345 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" Sep 11 18:38:05.036472 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.036444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.038981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.038960 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-238bae-serving-cert\"" Sep 11 18:38:05.049575 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.049551 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z"] Sep 11 18:38:05.072607 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.072576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:05.147614 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.147572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:05.217043 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.217008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73941f7b-2870-4a65-81af-f7dbcb5d6c34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.217221 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.217157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.217221 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.217190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.217317 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.217289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrgvs\" (UniqueName: \"kubernetes.io/projected/73941f7b-2870-4a65-81af-f7dbcb5d6c34-kube-api-access-lrgvs\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.318328 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.318241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.318328 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.318280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.318328 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.318319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrgvs\" (UniqueName: \"kubernetes.io/projected/73941f7b-2870-4a65-81af-f7dbcb5d6c34-kube-api-access-lrgvs\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.318544 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.318381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73941f7b-2870-4a65-81af-f7dbcb5d6c34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.318859 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:05.318837 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-serving-cert: secret "maas-api-e2e-shared-b-w2-238bae-serving-cert" not found Sep 11 18:38:05.318934 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:05.318915 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-maas-api-tls podName:73941f7b-2870-4a65-81af-f7dbcb5d6c34 nodeName:}" failed. No retries permitted until 2026-09-11 18:38:05.8188972 +0000 UTC m=+1354.045573543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-maas-api-tls") pod "maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" (UID: "73941f7b-2870-4a65-81af-f7dbcb5d6c34") : secret "maas-api-e2e-shared-b-w2-238bae-serving-cert" not found Sep 11 18:38:05.319223 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.319203 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73941f7b-2870-4a65-81af-f7dbcb5d6c34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.321531 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.321514 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.329699 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.329680 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrgvs\" (UniqueName: \"kubernetes.io/projected/73941f7b-2870-4a65-81af-f7dbcb5d6c34-kube-api-access-lrgvs\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.568957 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.568871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:05.673050 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.672989 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 11 18:38:05.673050 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.673043 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 11 18:38:05.673765 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.673737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:05.771321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.771275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:05.823772 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.823670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.827275 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.827234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:05.951594 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:05.951552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:06.140263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:06.140172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:06.261469 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:06.261444 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z"] Sep 11 18:38:06.263344 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:38:06.263316 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73941f7b_2870_4a65_81af_f7dbcb5d6c34.slice/crio-11934b948b1067496b25c1974035deac19ba42b2c3b7ca3d7b522a51d27d379b WatchSource:0}: Error finding container 11934b948b1067496b25c1974035deac19ba42b2c3b7ca3d7b522a51d27d379b: Status 404 returned error can't find the container with id 11934b948b1067496b25c1974035deac19ba42b2c3b7ca3d7b522a51d27d379b Sep 11 18:38:06.550501 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:06.550465 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" event={"ID":"73941f7b-2870-4a65-81af-f7dbcb5d6c34","Type":"ContainerStarted","Data":"11934b948b1067496b25c1974035deac19ba42b2c3b7ca3d7b522a51d27d379b"} Sep 11 18:38:06.672547 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:06.672513 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 11 18:38:06.672861 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:06.672568 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 11 18:38:07.057183 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:07.057141 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:07.558071 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:07.558032 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" event={"ID":"73941f7b-2870-4a65-81af-f7dbcb5d6c34","Type":"ContainerStarted","Data":"2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2"} Sep 11 18:38:07.558256 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:07.558167 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:07.579796 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:07.579752 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" podStartSLOduration=2.202923676 podStartE2EDuration="2.579737788s" totalImagesPullingTime="376.814112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:07.578343584 +0000 UTC m=+1355.805019944" watchObservedRunningTime="2026-09-11 18:38:07.579737788 +0000 UTC m=+1355.806414129" Sep 11 18:38:07.672310 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:07.672279 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 11 18:38:07.672527 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:07.672506 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 11 18:38:08.673270 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:08.673234 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 11 18:38:08.673672 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:08.673286 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 11 18:38:09.677037 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:09.677007 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:38:09.677408 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:09.677277 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:38:09.677989 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:09.677972 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:38:13.571457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:13.571424 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:38:14.721593 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:14.721554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:15.057374 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:15.057336 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:38:17.588920 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:17.588880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:18.808667 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:18.808626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:26.138027 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:26.137990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:26.676294 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:26.676250 2579 generic.go:356] "Generic (PLEG): container finished" podID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerID="0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a" exitCode=0 Sep 11 18:38:26.676505 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:26.676319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" event={"ID":"c6c95730-6b8e-47f5-8eac-c8c971da7722","Type":"ContainerDied","Data":"0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a"} Sep 11 18:38:26.676751 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:26.676737 2579 scope.go:117] "RemoveContainer" containerID="0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a" Sep 11 18:38:26.678029 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:26.678000 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:38:26.678113 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:26.678073 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:38:26.678113 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:26.678106 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:38:27.559691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:27.559651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:27.685928 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:27.685893 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" event={"ID":"c6c95730-6b8e-47f5-8eac-c8c971da7722","Type":"ContainerStarted","Data":"4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438"} Sep 11 18:38:28.038348 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:28.038303 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:38:28.039569 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:28.039537 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 18:38:28.039729 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:28.039609 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 18:38:28.439079 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:28.438969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:29.039015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:29.038973 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 18:38:29.039412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:29.039043 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 18:38:29.165666 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:29.165634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:29.494974 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:29.494870 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:38:29.497181 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:29.497149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:30.038429 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.038398 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 18:38:30.038665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.038455 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 18:38:30.039176 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.039146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:30.241131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.241101 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7816-665816"] Sep 11 18:38:30.241667 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.241652 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb58aa2-de56-4576-a8e5-bfc76d1c0164" containerName="istio-proxy" Sep 11 18:38:30.244991 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.244972 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-665816" Sep 11 18:38:30.247652 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.247621 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vvh9z\"" Sep 11 18:38:30.250551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.250526 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-665816"] Sep 11 18:38:30.372591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.372497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f289g\" (UniqueName: \"kubernetes.io/projected/9ea4d3f1-51c8-4443-8bd1-95b1133df920-kube-api-access-f289g\") pod \"test-curl-7816-665816\" (UID: \"9ea4d3f1-51c8-4443-8bd1-95b1133df920\") " pod="opendatahub/test-curl-7816-665816" Sep 11 18:38:30.473809 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.473776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f289g\" (UniqueName: \"kubernetes.io/projected/9ea4d3f1-51c8-4443-8bd1-95b1133df920-kube-api-access-f289g\") pod \"test-curl-7816-665816\" (UID: \"9ea4d3f1-51c8-4443-8bd1-95b1133df920\") " pod="opendatahub/test-curl-7816-665816" Sep 11 18:38:30.495295 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.495254 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f289g\" (UniqueName: \"kubernetes.io/projected/9ea4d3f1-51c8-4443-8bd1-95b1133df920-kube-api-access-f289g\") pod \"test-curl-7816-665816\" (UID: \"9ea4d3f1-51c8-4443-8bd1-95b1133df920\") " pod="opendatahub/test-curl-7816-665816" Sep 11 18:38:30.556944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.556905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-665816" Sep 11 18:38:30.852651 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.852611 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-665816"] Sep 11 18:38:30.853446 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:38:30.853414 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ea4d3f1_51c8_4443_8bd1_95b1133df920.slice/crio-75cdf27b56f3f0a7d9f114397dcc4075d9ff01dbb8fcf87493e9ff58c5ba0959 WatchSource:0}: Error finding container 75cdf27b56f3f0a7d9f114397dcc4075d9ff01dbb8fcf87493e9ff58c5ba0959: Status 404 returned error can't find the container with id 75cdf27b56f3f0a7d9f114397dcc4075d9ff01dbb8fcf87493e9ff58c5ba0959 Sep 11 18:38:30.940772 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:30.940697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:31.038647 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:31.038609 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 18:38:31.039070 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:31.038683 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 18:38:31.725751 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:31.725688 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-665816" event={"ID":"9ea4d3f1-51c8-4443-8bd1-95b1133df920","Type":"ContainerStarted","Data":"5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93"} Sep 11 18:38:31.726241 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:31.725760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-665816" event={"ID":"9ea4d3f1-51c8-4443-8bd1-95b1133df920","Type":"ContainerStarted","Data":"75cdf27b56f3f0a7d9f114397dcc4075d9ff01dbb8fcf87493e9ff58c5ba0959"} Sep 11 18:38:31.743588 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:31.743535 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7816-665816" podStartSLOduration=1.7435098359999999 podStartE2EDuration="1.743509836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:31.741235308 +0000 UTC m=+1379.967911648" watchObservedRunningTime="2026-09-11 18:38:31.743509836 +0000 UTC m=+1379.970186200" Sep 11 18:38:31.764720 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:31.764675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:32.038671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.038635 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 18:38:32.038877 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.038758 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 18:38:32.395406 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.395310 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7816-665816"] Sep 11 18:38:32.407130 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.407090 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7816-665816"] Sep 11 18:38:32.517025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.516977 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7816-910257"] Sep 11 18:38:32.517508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.517493 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea4d3f1-51c8-4443-8bd1-95b1133df920" containerName="test-curl-7816-665816" Sep 11 18:38:32.517554 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.517509 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea4d3f1-51c8-4443-8bd1-95b1133df920" containerName="test-curl-7816-665816" Sep 11 18:38:32.517589 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.517572 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea4d3f1-51c8-4443-8bd1-95b1133df920" containerName="test-curl-7816-665816" Sep 11 18:38:32.520483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.520457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:32.522205 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.522188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-910257" Sep 11 18:38:32.527840 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.527820 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-910257"] Sep 11 18:38:32.611301 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.611264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55qm\" (UniqueName: \"kubernetes.io/projected/7a35f837-1bb4-4347-bd33-4b73cc76cd59-kube-api-access-k55qm\") pod \"test-curl-7816-910257\" (UID: \"7a35f837-1bb4-4347-bd33-4b73cc76cd59\") " pod="opendatahub/test-curl-7816-910257" Sep 11 18:38:32.711979 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.711875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55qm\" (UniqueName: \"kubernetes.io/projected/7a35f837-1bb4-4347-bd33-4b73cc76cd59-kube-api-access-k55qm\") pod \"test-curl-7816-910257\" (UID: \"7a35f837-1bb4-4347-bd33-4b73cc76cd59\") " pod="opendatahub/test-curl-7816-910257" Sep 11 18:38:32.721070 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.721033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55qm\" (UniqueName: \"kubernetes.io/projected/7a35f837-1bb4-4347-bd33-4b73cc76cd59-kube-api-access-k55qm\") pod \"test-curl-7816-910257\" (UID: \"7a35f837-1bb4-4347-bd33-4b73cc76cd59\") " pod="opendatahub/test-curl-7816-910257" Sep 11 18:38:32.734931 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:32.734894 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7816-665816" container="test-curl-7816-665816" Sep 11 18:38:32.833995 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:32.833958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-910257" Sep 11 18:38:33.042447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:33.042414 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:38:33.043437 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:33.043416 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:38:33.113970 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:33.113923 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-910257"] Sep 11 18:38:33.115740 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:38:33.115659 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a35f837_1bb4_4347_bd33_4b73cc76cd59.slice/crio-384add7ab91f5dc3e49e5f030f42f3b4bf24dd5447fd402fca5af7b04be31cf0 WatchSource:0}: Error finding container 384add7ab91f5dc3e49e5f030f42f3b4bf24dd5447fd402fca5af7b04be31cf0: Status 404 returned error can't find the container with id 384add7ab91f5dc3e49e5f030f42f3b4bf24dd5447fd402fca5af7b04be31cf0 Sep 11 18:38:33.742439 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:33.742404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-910257" event={"ID":"7a35f837-1bb4-4347-bd33-4b73cc76cd59","Type":"ContainerStarted","Data":"3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097"} Sep 11 18:38:33.742439 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:33.742456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-910257" event={"ID":"7a35f837-1bb4-4347-bd33-4b73cc76cd59","Type":"ContainerStarted","Data":"384add7ab91f5dc3e49e5f030f42f3b4bf24dd5447fd402fca5af7b04be31cf0"} Sep 11 18:38:33.743109 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:33.742505 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7816-665816" podUID="9ea4d3f1-51c8-4443-8bd1-95b1133df920" containerName="test-curl-7816-665816" containerID="cri-o://5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93" gracePeriod=2 Sep 11 18:38:33.759952 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:33.759918 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7816-665816\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="9ea4d3f1-51c8-4443-8bd1-95b1133df920" pod="opendatahub/test-curl-7816-665816" Sep 11 18:38:33.760200 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:33.760160 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7816-910257" podStartSLOduration=1.760141255 podStartE2EDuration="1.760141255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:33.75789353 +0000 UTC m=+1381.984569869" watchObservedRunningTime="2026-09-11 18:38:33.760141255 +0000 UTC m=+1381.986817595" Sep 11 18:38:33.761669 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:33.761646 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7816-665816\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="9ea4d3f1-51c8-4443-8bd1-95b1133df920" pod="opendatahub/test-curl-7816-665816" Sep 11 18:38:34.395032 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.394937 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7816-910257"] Sep 11 18:38:34.398839 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.398811 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7816-910257"] Sep 11 18:38:34.510563 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.510534 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7816-688892"] Sep 11 18:38:34.510990 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.510978 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a35f837-1bb4-4347-bd33-4b73cc76cd59" containerName="test-curl-7816-910257" Sep 11 18:38:34.511051 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.510992 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a35f837-1bb4-4347-bd33-4b73cc76cd59" containerName="test-curl-7816-910257" Sep 11 18:38:34.511091 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.511077 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a35f837-1bb4-4347-bd33-4b73cc76cd59" containerName="test-curl-7816-910257" Sep 11 18:38:34.513892 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.513871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-688892" Sep 11 18:38:34.522985 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.522956 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-688892"] Sep 11 18:38:34.630760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.630719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/840f6447-35fb-449c-ac1e-1d99cb75f3a1-kube-api-access-f7qpm\") pod \"test-curl-7816-688892\" (UID: \"840f6447-35fb-449c-ac1e-1d99cb75f3a1\") " pod="opendatahub/test-curl-7816-688892" Sep 11 18:38:34.731901 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.731809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/840f6447-35fb-449c-ac1e-1d99cb75f3a1-kube-api-access-f7qpm\") pod \"test-curl-7816-688892\" (UID: \"840f6447-35fb-449c-ac1e-1d99cb75f3a1\") " pod="opendatahub/test-curl-7816-688892" Sep 11 18:38:34.740662 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.740634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/840f6447-35fb-449c-ac1e-1d99cb75f3a1-kube-api-access-f7qpm\") pod \"test-curl-7816-688892\" (UID: \"840f6447-35fb-449c-ac1e-1d99cb75f3a1\") " pod="opendatahub/test-curl-7816-688892" Sep 11 18:38:34.749934 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:34.749907 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7816-910257" container="test-curl-7816-910257" Sep 11 18:38:34.832044 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:34.832007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-688892" Sep 11 18:38:35.114182 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:35.114158 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-688892"] Sep 11 18:38:35.115806 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:38:35.115774 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod840f6447_35fb_449c_ac1e_1d99cb75f3a1.slice/crio-436ce65d4dd414674b05838c2700f6b6d5ada801aaefea26ffe99c596ecfecc6 WatchSource:0}: Error finding container 436ce65d4dd414674b05838c2700f6b6d5ada801aaefea26ffe99c596ecfecc6: Status 404 returned error can't find the container with id 436ce65d4dd414674b05838c2700f6b6d5ada801aaefea26ffe99c596ecfecc6 Sep 11 18:38:35.758601 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:35.758571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-688892" event={"ID":"840f6447-35fb-449c-ac1e-1d99cb75f3a1","Type":"ContainerStarted","Data":"0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b"} Sep 11 18:38:35.758916 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:35.758620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-688892" event={"ID":"840f6447-35fb-449c-ac1e-1d99cb75f3a1","Type":"ContainerStarted","Data":"436ce65d4dd414674b05838c2700f6b6d5ada801aaefea26ffe99c596ecfecc6"} Sep 11 18:38:35.758916 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:35.758758 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7816-910257" podUID="7a35f837-1bb4-4347-bd33-4b73cc76cd59" containerName="test-curl-7816-910257" containerID="cri-o://3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097" gracePeriod=2 Sep 11 18:38:35.775234 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:35.775193 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7816-910257\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="7a35f837-1bb4-4347-bd33-4b73cc76cd59" pod="opendatahub/test-curl-7816-910257" Sep 11 18:38:35.776695 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:35.776654 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7816-688892" podStartSLOduration=1.776638055 podStartE2EDuration="1.776638055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:35.773289016 +0000 UTC m=+1383.999965357" watchObservedRunningTime="2026-09-11 18:38:35.776638055 +0000 UTC m=+1384.003314394" Sep 11 18:38:35.924735 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:35.924697 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-665816" Sep 11 18:38:35.927123 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:35.927093 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7816-910257\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="7a35f837-1bb4-4347-bd33-4b73cc76cd59" pod="opendatahub/test-curl-7816-910257" Sep 11 18:38:36.046174 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.046138 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f289g\" (UniqueName: \"kubernetes.io/projected/9ea4d3f1-51c8-4443-8bd1-95b1133df920-kube-api-access-f289g\") pod \"9ea4d3f1-51c8-4443-8bd1-95b1133df920\" (UID: \"9ea4d3f1-51c8-4443-8bd1-95b1133df920\") " Sep 11 18:38:36.048179 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.048152 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea4d3f1-51c8-4443-8bd1-95b1133df920-kube-api-access-f289g" (OuterVolumeSpecName: "kube-api-access-f289g") pod "9ea4d3f1-51c8-4443-8bd1-95b1133df920" (UID: "9ea4d3f1-51c8-4443-8bd1-95b1133df920"). InnerVolumeSpecName "kube-api-access-f289g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:38:36.147189 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.147131 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f289g\" (UniqueName: \"kubernetes.io/projected/9ea4d3f1-51c8-4443-8bd1-95b1133df920-kube-api-access-f289g\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:36.229900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.229869 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea4d3f1-51c8-4443-8bd1-95b1133df920" path="/var/lib/kubelet/pods/9ea4d3f1-51c8-4443-8bd1-95b1133df920/volumes" Sep 11 18:38:36.374189 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.374105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7816-688892"] Sep 11 18:38:36.380803 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.380775 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7816-688892"] Sep 11 18:38:36.503306 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.503274 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7816-2573dd"] Sep 11 18:38:36.503775 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.503760 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840f6447-35fb-449c-ac1e-1d99cb75f3a1" containerName="test-curl-7816-688892" Sep 11 18:38:36.503839 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.503777 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="840f6447-35fb-449c-ac1e-1d99cb75f3a1" containerName="test-curl-7816-688892" Sep 11 18:38:36.503887 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.503856 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="840f6447-35fb-449c-ac1e-1d99cb75f3a1" containerName="test-curl-7816-688892" Sep 11 18:38:36.507185 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.507166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-2573dd" Sep 11 18:38:36.514999 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.514967 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-2573dd"] Sep 11 18:38:36.652662 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.652560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5x5\" (UniqueName: \"kubernetes.io/projected/e55b0cc8-3bab-464c-8847-cd40d51a9cee-kube-api-access-hl5x5\") pod \"test-curl-7816-2573dd\" (UID: \"e55b0cc8-3bab-464c-8847-cd40d51a9cee\") " pod="opendatahub/test-curl-7816-2573dd" Sep 11 18:38:36.753798 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.753759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5x5\" (UniqueName: \"kubernetes.io/projected/e55b0cc8-3bab-464c-8847-cd40d51a9cee-kube-api-access-hl5x5\") pod \"test-curl-7816-2573dd\" (UID: \"e55b0cc8-3bab-464c-8847-cd40d51a9cee\") " pod="opendatahub/test-curl-7816-2573dd" Sep 11 18:38:36.762671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.762642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5x5\" (UniqueName: \"kubernetes.io/projected/e55b0cc8-3bab-464c-8847-cd40d51a9cee-kube-api-access-hl5x5\") pod \"test-curl-7816-2573dd\" (UID: \"e55b0cc8-3bab-464c-8847-cd40d51a9cee\") " pod="opendatahub/test-curl-7816-2573dd" Sep 11 18:38:36.766533 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.766498 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ea4d3f1-51c8-4443-8bd1-95b1133df920" containerID="5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93" exitCode=137 Sep 11 18:38:36.766671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.766536 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-665816" Sep 11 18:38:36.766671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.766597 2579 scope.go:117] "RemoveContainer" containerID="5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93" Sep 11 18:38:36.766898 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:36.766882 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7816-688892" container="test-curl-7816-688892" Sep 11 18:38:36.778334 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.778316 2579 scope.go:117] "RemoveContainer" containerID="5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93" Sep 11 18:38:36.778614 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:36.778594 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93\": container with ID starting with 5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93 not found: ID does not exist" containerID="5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93" Sep 11 18:38:36.778659 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.778624 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93"} err="failed to get container status \"5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93\": rpc error: code = NotFound desc = could not find container \"5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93\": container with ID starting with 5a49dfef6375d0abbc2b20570976987b09a92d84a8d8241b9e4b34d331daba93 not found: ID does not exist" Sep 11 18:38:36.818386 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:36.818355 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-2573dd" Sep 11 18:38:37.092351 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:37.092323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-2573dd"] Sep 11 18:38:37.094487 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:38:37.094461 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode55b0cc8_3bab_464c_8847_cd40d51a9cee.slice/crio-d18ab5d6f4df3f993639ed87897c2606833632b95cf886319bb9f920138138eb WatchSource:0}: Error finding container d18ab5d6f4df3f993639ed87897c2606833632b95cf886319bb9f920138138eb: Status 404 returned error can't find the container with id d18ab5d6f4df3f993639ed87897c2606833632b95cf886319bb9f920138138eb Sep 11 18:38:37.777439 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:37.777405 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-2573dd" event={"ID":"e55b0cc8-3bab-464c-8847-cd40d51a9cee","Type":"ContainerStarted","Data":"dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51"} Sep 11 18:38:37.777901 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:37.777870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-2573dd" event={"ID":"e55b0cc8-3bab-464c-8847-cd40d51a9cee","Type":"ContainerStarted","Data":"d18ab5d6f4df3f993639ed87897c2606833632b95cf886319bb9f920138138eb"} Sep 11 18:38:37.778517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:37.778492 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7816-688892" podUID="840f6447-35fb-449c-ac1e-1d99cb75f3a1" containerName="test-curl-7816-688892" containerID="cri-o://0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b" gracePeriod=2 Sep 11 18:38:37.795574 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:37.795540 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7816-688892\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="840f6447-35fb-449c-ac1e-1d99cb75f3a1" pod="opendatahub/test-curl-7816-688892" Sep 11 18:38:37.796353 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:37.796313 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7816-2573dd" podStartSLOduration=1.7962983399999999 podStartE2EDuration="1.79629834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:37.793489193 +0000 UTC m=+1386.020165537" watchObservedRunningTime="2026-09-11 18:38:37.79629834 +0000 UTC m=+1386.022974722" Sep 11 18:38:37.933275 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:37.933241 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-910257" Sep 11 18:38:37.935773 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:37.935739 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7816-688892\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="840f6447-35fb-449c-ac1e-1d99cb75f3a1" pod="opendatahub/test-curl-7816-688892" Sep 11 18:38:38.068663 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.068583 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k55qm\" (UniqueName: \"kubernetes.io/projected/7a35f837-1bb4-4347-bd33-4b73cc76cd59-kube-api-access-k55qm\") pod \"7a35f837-1bb4-4347-bd33-4b73cc76cd59\" (UID: \"7a35f837-1bb4-4347-bd33-4b73cc76cd59\") " Sep 11 18:38:38.070493 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.070465 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a35f837-1bb4-4347-bd33-4b73cc76cd59-kube-api-access-k55qm" (OuterVolumeSpecName: "kube-api-access-k55qm") pod "7a35f837-1bb4-4347-bd33-4b73cc76cd59" (UID: "7a35f837-1bb4-4347-bd33-4b73cc76cd59"). InnerVolumeSpecName "kube-api-access-k55qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:38:38.169697 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.169665 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k55qm\" (UniqueName: \"kubernetes.io/projected/7a35f837-1bb4-4347-bd33-4b73cc76cd59-kube-api-access-k55qm\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:38.229238 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.229208 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a35f837-1bb4-4347-bd33-4b73cc76cd59" path="/var/lib/kubelet/pods/7a35f837-1bb4-4347-bd33-4b73cc76cd59/volumes" Sep 11 18:38:38.402355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.402270 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7816-2573dd"] Sep 11 18:38:38.426304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.426276 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7816-2573dd"] Sep 11 18:38:38.787318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.787279 2579 generic.go:356] "Generic (PLEG): container finished" podID="7a35f837-1bb4-4347-bd33-4b73cc76cd59" containerID="3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097" exitCode=137 Sep 11 18:38:38.787318 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.787317 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-910257" Sep 11 18:38:38.787901 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.787368 2579 scope.go:117] "RemoveContainer" containerID="3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097" Sep 11 18:38:38.787901 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:38.787594 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7816-2573dd" container="test-curl-7816-2573dd" Sep 11 18:38:38.797549 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.797525 2579 scope.go:117] "RemoveContainer" containerID="3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097" Sep 11 18:38:38.797869 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:38.797846 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097\": container with ID starting with 3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097 not found: ID does not exist" containerID="3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097" Sep 11 18:38:38.797936 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:38.797879 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097"} err="failed to get container status \"3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097\": rpc error: code = NotFound desc = could not find container \"3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097\": container with ID starting with 3227e883a16aefa536556ae8991a2d99ed41c64512515f85663f47f004c9d097 not found: ID does not exist" Sep 11 18:38:39.066395 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.066310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:39.330954 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.330868 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7"] Sep 11 18:38:39.331441 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.331422 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55b0cc8-3bab-464c-8847-cd40d51a9cee" containerName="test-curl-7816-2573dd" Sep 11 18:38:39.331532 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.331444 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55b0cc8-3bab-464c-8847-cd40d51a9cee" containerName="test-curl-7816-2573dd" Sep 11 18:38:39.331604 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.331590 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55b0cc8-3bab-464c-8847-cd40d51a9cee" containerName="test-curl-7816-2573dd" Sep 11 18:38:39.336128 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.336104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.338837 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.338809 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-699caf96-openshift-default-dockercfg-vsnp9\"" Sep 11 18:38:39.346408 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.346332 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7"] Sep 11 18:38:39.367678 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.367650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:39.484480 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.484434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d5277-d4b5-46b4-a216-472e13a7c580-istiod-ca-cert\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.484693 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.484646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-socket\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.484693 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.484685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-envoy\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.485059 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.485037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-podinfo\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.485160 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.485081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-credential-socket\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.485227 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.485180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-data\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.485227 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.485209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-token\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.485336 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.485253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwgvd\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-kube-api-access-zwgvd\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.485336 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.485321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-certs\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586341 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-data\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586341 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-token\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586341 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwgvd\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-kube-api-access-zwgvd\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-certs\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586721 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d5277-d4b5-46b4-a216-472e13a7c580-istiod-ca-cert\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586721 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-data\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586851 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-socket\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586903 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-envoy\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-podinfo\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.586960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.586937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-credential-socket\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.587075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.587051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-certs\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.587257 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.587237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-credential-socket\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.587365 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.587342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d5277-d4b5-46b4-a216-472e13a7c580-istiod-ca-cert\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.587720 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.587679 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-socket\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.589671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.589640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-envoy\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.590026 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.590001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-podinfo\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.602264 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.602243 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-token\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.608792 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.608770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwgvd\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-kube-api-access-zwgvd\") pod \"e2e-mt-699caf96-openshift-default-847c959b56-mfqr7\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.646606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.646574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:39.646764 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.646617 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:39.795752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.795719 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7816-2573dd" podUID="e55b0cc8-3bab-464c-8847-cd40d51a9cee" containerName="test-curl-7816-2573dd" containerID="cri-o://dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51" gracePeriod=2 Sep 11 18:38:39.797878 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:39.797836 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7816-2573dd\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="e55b0cc8-3bab-464c-8847-cd40d51a9cee" pod="opendatahub/test-curl-7816-2573dd" Sep 11 18:38:39.932372 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.932346 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7"] Sep 11 18:38:39.933158 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:38:39.933132 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05d5277_d4b5_46b4_a216_472e13a7c580.slice/crio-8bf4c9e2fc409411853c4342e17ee88b1e8fa6dd332468d926beae7c766adf59 WatchSource:0}: Error finding container 8bf4c9e2fc409411853c4342e17ee88b1e8fa6dd332468d926beae7c766adf59: Status 404 returned error can't find the container with id 8bf4c9e2fc409411853c4342e17ee88b1e8fa6dd332468d926beae7c766adf59 Sep 11 18:38:39.935535 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.935503 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:38:39.935610 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.935573 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:38:39.935610 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.935601 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:38:39.955008 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:39.954987 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-688892" Sep 11 18:38:39.957140 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:39.957112 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7816-2573dd\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="e55b0cc8-3bab-464c-8847-cd40d51a9cee" pod="opendatahub/test-curl-7816-2573dd" Sep 11 18:38:40.093575 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.093536 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/840f6447-35fb-449c-ac1e-1d99cb75f3a1-kube-api-access-f7qpm\") pod \"840f6447-35fb-449c-ac1e-1d99cb75f3a1\" (UID: \"840f6447-35fb-449c-ac1e-1d99cb75f3a1\") " Sep 11 18:38:40.096214 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.096156 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840f6447-35fb-449c-ac1e-1d99cb75f3a1-kube-api-access-f7qpm" (OuterVolumeSpecName: "kube-api-access-f7qpm") pod "840f6447-35fb-449c-ac1e-1d99cb75f3a1" (UID: "840f6447-35fb-449c-ac1e-1d99cb75f3a1"). InnerVolumeSpecName "kube-api-access-f7qpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:38:40.195509 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.195424 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qpm\" (UniqueName: \"kubernetes.io/projected/840f6447-35fb-449c-ac1e-1d99cb75f3a1-kube-api-access-f7qpm\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:40.230747 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.230700 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="840f6447-35fb-449c-ac1e-1d99cb75f3a1" path="/var/lib/kubelet/pods/840f6447-35fb-449c-ac1e-1d99cb75f3a1/volumes" Sep 11 18:38:40.804038 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.804003 2579 generic.go:356] "Generic (PLEG): container finished" podID="840f6447-35fb-449c-ac1e-1d99cb75f3a1" containerID="0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b" exitCode=137 Sep 11 18:38:40.804501 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.804045 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-688892" Sep 11 18:38:40.804501 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.804100 2579 scope.go:117] "RemoveContainer" containerID="0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b" Sep 11 18:38:40.806454 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.806189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" event={"ID":"d05d5277-d4b5-46b4-a216-472e13a7c580","Type":"ContainerStarted","Data":"de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047"} Sep 11 18:38:40.806454 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.806248 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" event={"ID":"d05d5277-d4b5-46b4-a216-472e13a7c580","Type":"ContainerStarted","Data":"8bf4c9e2fc409411853c4342e17ee88b1e8fa6dd332468d926beae7c766adf59"} Sep 11 18:38:40.813246 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.813230 2579 scope.go:117] "RemoveContainer" containerID="0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b" Sep 11 18:38:40.813508 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:40.813487 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b\": container with ID starting with 0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b not found: ID does not exist" containerID="0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b" Sep 11 18:38:40.813579 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.813515 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b"} err="failed to get container status \"0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b\": rpc error: code = NotFound desc = could not find container \"0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b\": container with ID starting with 0e1c38a5ecbd60db722af32fc2dddb128644f23af1791c1e2ad4edf04fd3666b not found: ID does not exist" Sep 11 18:38:40.857556 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:40.857425 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" podStartSLOduration=1.857402407 podStartE2EDuration="1.857402407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:40.852412917 +0000 UTC m=+1389.079089260" watchObservedRunningTime="2026-09-11 18:38:40.857402407 +0000 UTC m=+1389.084078749" Sep 11 18:38:41.646984 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:41.646950 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:41.653219 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:41.653186 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:41.813999 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:41.813973 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:41.815098 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:41.815079 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:38:41.989267 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:41.989240 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-2573dd" Sep 11 18:38:42.011093 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.011065 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl5x5\" (UniqueName: \"kubernetes.io/projected/e55b0cc8-3bab-464c-8847-cd40d51a9cee-kube-api-access-hl5x5\") pod \"e55b0cc8-3bab-464c-8847-cd40d51a9cee\" (UID: \"e55b0cc8-3bab-464c-8847-cd40d51a9cee\") " Sep 11 18:38:42.013061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.013034 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55b0cc8-3bab-464c-8847-cd40d51a9cee-kube-api-access-hl5x5" (OuterVolumeSpecName: "kube-api-access-hl5x5") pod "e55b0cc8-3bab-464c-8847-cd40d51a9cee" (UID: "e55b0cc8-3bab-464c-8847-cd40d51a9cee"). InnerVolumeSpecName "kube-api-access-hl5x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:38:42.112262 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.112225 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl5x5\" (UniqueName: \"kubernetes.io/projected/e55b0cc8-3bab-464c-8847-cd40d51a9cee-kube-api-access-hl5x5\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:38:42.230468 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.230394 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e55b0cc8-3bab-464c-8847-cd40d51a9cee" path="/var/lib/kubelet/pods/e55b0cc8-3bab-464c-8847-cd40d51a9cee/volumes" Sep 11 18:38:42.661661 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.661627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:42.822065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.822028 2579 generic.go:356] "Generic (PLEG): container finished" podID="e55b0cc8-3bab-464c-8847-cd40d51a9cee" containerID="dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51" exitCode=137 Sep 11 18:38:42.822494 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.822071 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-2573dd" Sep 11 18:38:42.822494 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.822129 2579 scope.go:117] "RemoveContainer" containerID="dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51" Sep 11 18:38:42.835090 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.835068 2579 scope.go:117] "RemoveContainer" containerID="dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51" Sep 11 18:38:42.835426 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:42.835406 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51\": container with ID starting with dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51 not found: ID does not exist" containerID="dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51" Sep 11 18:38:42.835481 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.835440 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51"} err="failed to get container status \"dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51\": rpc error: code = NotFound desc = could not find container \"dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51\": container with ID starting with dcc23d1164eea70c056733b3de6d22994fcd4137b11980c8a67effff94633d51 not found: ID does not exist" Sep 11 18:38:42.863010 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.862961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:42.972638 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:42.972522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:43.943586 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:43.943548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:44.802927 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:44.802878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:45.708498 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:45.708458 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:46.431662 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:46.431627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:47.184508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:47.184471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:48.103046 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.103016 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc"] Sep 11 18:38:48.108132 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.108109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.114154 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.114131 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-699caf96-serving-cert\"" Sep 11 18:38:48.146178 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.146148 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc"] Sep 11 18:38:48.167627 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.167594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9pq\" (UniqueName: \"kubernetes.io/projected/acba202b-845c-4a2e-bb3b-6f6e009bce05-kube-api-access-vw9pq\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.167798 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.167644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-maas-api-tls\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.167798 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.167773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-metrics-certs\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.167892 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.167823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acba202b-845c-4a2e-bb3b-6f6e009bce05-openshift-service-ca\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.205934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.205899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:48.269176 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.269143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9pq\" (UniqueName: \"kubernetes.io/projected/acba202b-845c-4a2e-bb3b-6f6e009bce05-kube-api-access-vw9pq\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.269364 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.269193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-maas-api-tls\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.269364 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.269238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-metrics-certs\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.269364 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.269266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acba202b-845c-4a2e-bb3b-6f6e009bce05-openshift-service-ca\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.269548 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:48.269366 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-serving-cert: secret "maas-api-e2e-mt-699caf96-serving-cert" not found Sep 11 18:38:48.269548 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:38:48.269427 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-maas-api-tls podName:acba202b-845c-4a2e-bb3b-6f6e009bce05 nodeName:}" failed. No retries permitted until 2026-09-11 18:38:48.769410492 +0000 UTC m=+1396.996086842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-maas-api-tls") pod "maas-api-e2e-mt-699caf96-577cdf579-9xptc" (UID: "acba202b-845c-4a2e-bb3b-6f6e009bce05") : secret "maas-api-e2e-mt-699caf96-serving-cert" not found Sep 11 18:38:48.269896 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.269873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acba202b-845c-4a2e-bb3b-6f6e009bce05-openshift-service-ca\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.271606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.271586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-metrics-certs\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.287931 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.287903 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9pq\" (UniqueName: \"kubernetes.io/projected/acba202b-845c-4a2e-bb3b-6f6e009bce05-kube-api-access-vw9pq\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.318909 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.318880 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:38:48.352608 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.352570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:48.559164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.559131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:48.747547 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.747512 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:48.774235 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.774198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-maas-api-tls\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.776830 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.776802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-maas-api-tls\") pod \"maas-api-e2e-mt-699caf96-577cdf579-9xptc\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:48.848225 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:48.848132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:49.018826 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:49.018783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:49.296932 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:49.296894 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc"] Sep 11 18:38:49.298328 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:38:49.298299 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacba202b_845c_4a2e_bb3b_6f6e009bce05.slice/crio-eb8b23cdb0b1e2e4fbeb9a1ea2feccfdf23515c71adac636074359652733b5c1 WatchSource:0}: Error finding container eb8b23cdb0b1e2e4fbeb9a1ea2feccfdf23515c71adac636074359652733b5c1: Status 404 returned error can't find the container with id eb8b23cdb0b1e2e4fbeb9a1ea2feccfdf23515c71adac636074359652733b5c1 Sep 11 18:38:49.869448 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:49.869408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" event={"ID":"acba202b-845c-4a2e-bb3b-6f6e009bce05","Type":"ContainerStarted","Data":"19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9"} Sep 11 18:38:49.869643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:49.869478 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" event={"ID":"acba202b-845c-4a2e-bb3b-6f6e009bce05","Type":"ContainerStarted","Data":"eb8b23cdb0b1e2e4fbeb9a1ea2feccfdf23515c71adac636074359652733b5c1"} Sep 11 18:38:49.869643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:49.869586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:49.898573 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:49.898522 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" podStartSLOduration=1.4143790059999999 podStartE2EDuration="1.898505115s" totalImagesPullingTime="484.126109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:49.896805722 +0000 UTC m=+1398.123482092" watchObservedRunningTime="2026-09-11 18:38:49.898505115 +0000 UTC m=+1398.125181454" Sep 11 18:38:54.988288 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:54.988246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:55.879718 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:55.879682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:55.882313 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:55.882277 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:38:56.847145 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:56.847112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:57.386179 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:57.386141 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:57.608475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:57.608437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:58.433068 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:58.433028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:38:59.353523 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:38:59.353483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:00.128402 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:00.128369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:00.941591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:00.941555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:01.786022 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:01.785986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:02.435753 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:02.435716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:02.598312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:02.598273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:02.698226 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:02.698129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:03.519024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:03.518986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:04.265776 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:04.265733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:05.063920 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:05.063881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:06.072638 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:06.072591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:06.832026 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:06.831993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:07.565864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:07.565822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:07.660509 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:07.660470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:07.760832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:07.760797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:07.842888 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:07.842801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:07.913522 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:07.913489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:08.209608 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:08.209504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:08.308569 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:08.308523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:08.939071 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:08.939043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:09.834431 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:09.834393 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:10.644974 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:10.644931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:11.375248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:11.375211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:12.128873 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:12.128842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:13.053757 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:13.053699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:13.274242 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:13.274207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:17.071926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:17.071889 2579 generic.go:356] "Generic (PLEG): container finished" podID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerID="de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047" exitCode=0 Sep 11 18:39:17.072348 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:17.071961 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" event={"ID":"d05d5277-d4b5-46b4-a216-472e13a7c580","Type":"ContainerDied","Data":"de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047"} Sep 11 18:39:17.072420 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:17.072405 2579 scope.go:117] "RemoveContainer" containerID="de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047" Sep 11 18:39:17.073597 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:17.073568 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:39:17.073692 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:17.073640 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:39:17.073692 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:17.073671 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:39:18.079739 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:18.079687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" event={"ID":"d05d5277-d4b5-46b4-a216-472e13a7c580","Type":"ContainerStarted","Data":"16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da"} Sep 11 18:39:18.647667 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:18.647629 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:39:18.647904 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:18.647880 2579 patch_prober.go:28] interesting pod/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 18:39:18.647950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:18.647932 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 18:39:19.647209 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:19.647172 2579 patch_prober.go:28] interesting pod/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 18:39:19.647608 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:19.647226 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 18:39:19.969898 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:19.969815 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:39:20.648019 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:20.647982 2579 patch_prober.go:28] interesting pod/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 18:39:20.648515 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:20.648042 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 18:39:21.647541 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:21.647497 2579 patch_prober.go:28] interesting pod/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 18:39:21.647748 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:21.647554 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 18:39:22.647008 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:22.646968 2579 patch_prober.go:28] interesting pod/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 18:39:22.647455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:22.647022 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 18:39:23.647147 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:23.647109 2579 patch_prober.go:28] interesting pod/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 18:39:23.647545 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:23.647167 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 18:39:24.651311 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:24.651238 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:39:24.651662 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:24.651585 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:39:24.652393 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:24.652377 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:39:25.897014 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:25.896969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:34.494107 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:34.494072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:50.354418 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:50.354381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:50.592426 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:50.592389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:51.183247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:51.183207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:52.007003 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:52.006967 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:52.925760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:52.925722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:53.970186 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:53.970150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:54.217909 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:54.217874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:54.882601 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:54.882565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:55.699974 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:55.699929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:56.535449 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:56.535415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:56.974659 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:56.974581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:39:57.306003 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:39:57.305964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:01.510075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.510042 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh"] Sep 11 18:40:01.510431 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.510292 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" podUID="2b16857b-a659-4549-99b8-74f9cf1f9353" containerName="maas-api" containerID="cri-o://cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1" gracePeriod=30 Sep 11 18:40:01.562425 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.562394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:01.664622 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.664514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:01.765720 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.765633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:01.777173 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.777152 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:40:01.843850 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.843818 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-maas-api-tls\") pod \"2b16857b-a659-4549-99b8-74f9cf1f9353\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " Sep 11 18:40:01.844042 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.843871 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-metrics-certs\") pod \"2b16857b-a659-4549-99b8-74f9cf1f9353\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " Sep 11 18:40:01.844042 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.843926 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcdjz\" (UniqueName: \"kubernetes.io/projected/2b16857b-a659-4549-99b8-74f9cf1f9353-kube-api-access-jcdjz\") pod \"2b16857b-a659-4549-99b8-74f9cf1f9353\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " Sep 11 18:40:01.844175 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.844091 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b16857b-a659-4549-99b8-74f9cf1f9353-openshift-service-ca\") pod \"2b16857b-a659-4549-99b8-74f9cf1f9353\" (UID: \"2b16857b-a659-4549-99b8-74f9cf1f9353\") " Sep 11 18:40:01.844499 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.844461 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b16857b-a659-4549-99b8-74f9cf1f9353-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b16857b-a659-4549-99b8-74f9cf1f9353" (UID: "2b16857b-a659-4549-99b8-74f9cf1f9353"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:40:01.846144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.846119 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b16857b-a659-4549-99b8-74f9cf1f9353" (UID: "2b16857b-a659-4549-99b8-74f9cf1f9353"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:40:01.846523 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.846502 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b16857b-a659-4549-99b8-74f9cf1f9353" (UID: "2b16857b-a659-4549-99b8-74f9cf1f9353"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:40:01.846626 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.846546 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b16857b-a659-4549-99b8-74f9cf1f9353-kube-api-access-jcdjz" (OuterVolumeSpecName: "kube-api-access-jcdjz") pod "2b16857b-a659-4549-99b8-74f9cf1f9353" (UID: "2b16857b-a659-4549-99b8-74f9cf1f9353"). InnerVolumeSpecName "kube-api-access-jcdjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:40:01.865409 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.865377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:01.945665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.945622 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcdjz\" (UniqueName: \"kubernetes.io/projected/2b16857b-a659-4549-99b8-74f9cf1f9353-kube-api-access-jcdjz\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:01.945665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.945655 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b16857b-a659-4549-99b8-74f9cf1f9353-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:01.945665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.945669 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:01.945915 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:01.945681 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b16857b-a659-4549-99b8-74f9cf1f9353-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:02.157153 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.157069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:02.256483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.256454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:02.367480 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.367444 2579 generic.go:356] "Generic (PLEG): container finished" podID="2b16857b-a659-4549-99b8-74f9cf1f9353" containerID="cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1" exitCode=0 Sep 11 18:40:02.367649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.367485 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" Sep 11 18:40:02.367649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.367540 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" event={"ID":"2b16857b-a659-4549-99b8-74f9cf1f9353","Type":"ContainerDied","Data":"cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1"} Sep 11 18:40:02.367649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.367587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh" event={"ID":"2b16857b-a659-4549-99b8-74f9cf1f9353","Type":"ContainerDied","Data":"80e968b7eb2f501fbfaaa50911185fcfd86b04ff10f96bb1b1c726bb7415ff68"} Sep 11 18:40:02.367649 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.367609 2579 scope.go:117] "RemoveContainer" containerID="cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1" Sep 11 18:40:02.377296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.377278 2579 scope.go:117] "RemoveContainer" containerID="cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1" Sep 11 18:40:02.377600 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:40:02.377574 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1\": container with ID starting with cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1 not found: ID does not exist" containerID="cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1" Sep 11 18:40:02.377758 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.377605 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1"} err="failed to get container status \"cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1\": rpc error: code = NotFound desc = could not find container \"cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1\": container with ID starting with cde21e5d6dccac30e5e5ca16d30eedcb94e43b41cc064cbf9e8e1ea2f265fbf1 not found: ID does not exist" Sep 11 18:40:02.393118 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.393090 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh"] Sep 11 18:40:02.402376 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:02.402349 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8f201-56576846c9-fmkvh"] Sep 11 18:40:03.032320 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:03.032281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:04.073664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:04.073624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:04.230261 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:04.230220 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b16857b-a659-4549-99b8-74f9cf1f9353" path="/var/lib/kubelet/pods/2b16857b-a659-4549-99b8-74f9cf1f9353/volumes" Sep 11 18:40:04.830899 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:04.830859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:05.751894 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:05.751848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:06.395371 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:06.395332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:06.528478 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:06.528441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:07.243601 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:07.243565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:07.575187 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:07.575151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:07.808673 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:07.808634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:07.936759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:07.936643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:09.654355 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:09.654311 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:40:12.957759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:12.957720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:21.385363 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:21.385300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:32.342073 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:32.342047 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:40:32.346426 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:32.346399 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:40:44.356964 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:44.356925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:44.591761 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:44.591694 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4"] Sep 11 18:40:44.591998 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:44.591947 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" containerID="cri-o://45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068" gracePeriod=30 Sep 11 18:40:44.622255 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:44.622155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:45.162670 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:45.162628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:45.562629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:45.562595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:45.692913 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:45.692874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:46.871282 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:46.871243 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:40:47.586913 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:47.586874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:48.743714 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:48.743675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:49.130072 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.130037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:49.228366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.228321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:49.590627 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.590597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:49.860194 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.860122 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:40:49.932725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.932665 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-istio-token\") pod \"82f015fd-0119-494c-96bc-8726b34166ad\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " Sep 11 18:40:49.932725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.932732 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f015fd-0119-494c-96bc-8726b34166ad-istio-podinfo\") pod \"82f015fd-0119-494c-96bc-8726b34166ad\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " Sep 11 18:40:49.932981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.932773 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-certs\") pod \"82f015fd-0119-494c-96bc-8726b34166ad\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " Sep 11 18:40:49.932981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.932819 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-credential-socket\") pod \"82f015fd-0119-494c-96bc-8726b34166ad\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " Sep 11 18:40:49.932981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.932845 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f015fd-0119-494c-96bc-8726b34166ad-istiod-ca-cert\") pod \"82f015fd-0119-494c-96bc-8726b34166ad\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " Sep 11 18:40:49.932981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.932878 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wldlx\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-kube-api-access-wldlx\") pod \"82f015fd-0119-494c-96bc-8726b34166ad\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " Sep 11 18:40:49.932981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.932921 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-data\") pod \"82f015fd-0119-494c-96bc-8726b34166ad\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " Sep 11 18:40:49.932981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.932956 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-envoy\") pod \"82f015fd-0119-494c-96bc-8726b34166ad\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " Sep 11 18:40:49.933294 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.932996 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-socket\") pod \"82f015fd-0119-494c-96bc-8726b34166ad\" (UID: \"82f015fd-0119-494c-96bc-8726b34166ad\") " Sep 11 18:40:49.933294 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.933119 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82f015fd-0119-494c-96bc-8726b34166ad" (UID: "82f015fd-0119-494c-96bc-8726b34166ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:40:49.933294 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.933108 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82f015fd-0119-494c-96bc-8726b34166ad" (UID: "82f015fd-0119-494c-96bc-8726b34166ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:40:49.933455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.933320 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "82f015fd-0119-494c-96bc-8726b34166ad" (UID: "82f015fd-0119-494c-96bc-8726b34166ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:40:49.933455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.933315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f015fd-0119-494c-96bc-8726b34166ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82f015fd-0119-494c-96bc-8726b34166ad" (UID: "82f015fd-0119-494c-96bc-8726b34166ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:40:49.933455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.933377 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82f015fd-0119-494c-96bc-8726b34166ad" (UID: "82f015fd-0119-494c-96bc-8726b34166ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:40:49.933455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.933399 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:49.933455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.933420 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-credential-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:49.935097 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.935069 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "82f015fd-0119-494c-96bc-8726b34166ad" (UID: "82f015fd-0119-494c-96bc-8726b34166ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:40:49.935212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.935109 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82f015fd-0119-494c-96bc-8726b34166ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82f015fd-0119-494c-96bc-8726b34166ad" (UID: "82f015fd-0119-494c-96bc-8726b34166ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:40:49.935212 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.935116 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82f015fd-0119-494c-96bc-8726b34166ad" (UID: "82f015fd-0119-494c-96bc-8726b34166ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:40:49.935292 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:49.935218 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-kube-api-access-wldlx" (OuterVolumeSpecName: "kube-api-access-wldlx") pod "82f015fd-0119-494c-96bc-8726b34166ad" (UID: "82f015fd-0119-494c-96bc-8726b34166ad"). InnerVolumeSpecName "kube-api-access-wldlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:40:50.034965 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.034927 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-istio-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:50.034965 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.034958 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f015fd-0119-494c-96bc-8726b34166ad-istio-podinfo\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:50.034965 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.034968 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f015fd-0119-494c-96bc-8726b34166ad-istiod-ca-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:50.035197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.034977 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wldlx\" (UniqueName: \"kubernetes.io/projected/82f015fd-0119-494c-96bc-8726b34166ad-kube-api-access-wldlx\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:50.035197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.034987 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-data\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:50.035197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.034995 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-istio-envoy\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:50.035197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.035003 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f015fd-0119-494c-96bc-8726b34166ad-workload-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:40:50.443466 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.443427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:50.688247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.688208 2579 generic.go:356] "Generic (PLEG): container finished" podID="82f015fd-0119-494c-96bc-8726b34166ad" containerID="45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068" exitCode=0 Sep 11 18:40:50.688445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.688285 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" Sep 11 18:40:50.688445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.688291 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" event={"ID":"82f015fd-0119-494c-96bc-8726b34166ad","Type":"ContainerDied","Data":"45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068"} Sep 11 18:40:50.688445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.688392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4" event={"ID":"82f015fd-0119-494c-96bc-8726b34166ad","Type":"ContainerDied","Data":"6c05637af05f80ae6809ef656d6b00969590085748e97b82ac8733ea64451e40"} Sep 11 18:40:50.688445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.688425 2579 scope.go:117] "RemoveContainer" containerID="45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068" Sep 11 18:40:50.698404 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.698380 2579 scope.go:117] "RemoveContainer" containerID="32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597" Sep 11 18:40:50.707093 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.707073 2579 scope.go:117] "RemoveContainer" containerID="45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068" Sep 11 18:40:50.707380 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:40:50.707359 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068\": container with ID starting with 45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068 not found: ID does not exist" containerID="45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068" Sep 11 18:40:50.707458 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.707393 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068"} err="failed to get container status \"45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068\": rpc error: code = NotFound desc = could not find container \"45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068\": container with ID starting with 45ce5bd8c1db47ec578765234bac72cc7d5097d07cb25167f5a9786e4ba55068 not found: ID does not exist" Sep 11 18:40:50.707458 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.707419 2579 scope.go:117] "RemoveContainer" containerID="32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597" Sep 11 18:40:50.707691 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:40:50.707673 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597\": container with ID starting with 32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597 not found: ID does not exist" containerID="32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597" Sep 11 18:40:50.707780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.707720 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597"} err="failed to get container status \"32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597\": rpc error: code = NotFound desc = could not find container \"32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597\": container with ID starting with 32e12e2e6bb7e558170b80f6745469d5cda04b03bc00f1029f75af7094b6b597 not found: ID does not exist" Sep 11 18:40:50.732481 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.732447 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4"] Sep 11 18:40:50.751353 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:50.751324 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e8f201-openshift-default-7fc967d568-shbl4"] Sep 11 18:40:51.278673 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:51.278636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:51.427321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:51.427290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:51.968550 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:51.968489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:52.231509 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:52.231427 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f015fd-0119-494c-96bc-8726b34166ad" path="/var/lib/kubelet/pods/82f015fd-0119-494c-96bc-8726b34166ad/volumes" Sep 11 18:40:52.236614 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:52.236579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:52.656421 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:52.656386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:53.221858 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.221821 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn"] Sep 11 18:40:53.222609 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.222582 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" Sep 11 18:40:53.222609 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.222607 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" Sep 11 18:40:53.222825 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.222626 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b16857b-a659-4549-99b8-74f9cf1f9353" containerName="maas-api" Sep 11 18:40:53.222825 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.222636 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b16857b-a659-4549-99b8-74f9cf1f9353" containerName="maas-api" Sep 11 18:40:53.222825 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.222741 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b16857b-a659-4549-99b8-74f9cf1f9353" containerName="maas-api" Sep 11 18:40:53.222825 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.222749 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" Sep 11 18:40:53.227842 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.227808 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.233321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.233300 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-d90e71-serving-cert\"" Sep 11 18:40:53.238937 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.238916 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn"] Sep 11 18:40:53.269686 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.269649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.269880 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.269729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz76r\" (UniqueName: \"kubernetes.io/projected/1155c028-9680-4af3-a548-0b4bbca1d763-kube-api-access-mz76r\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.269880 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.269814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.269880 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.269863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1155c028-9680-4af3-a548-0b4bbca1d763-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.270042 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.269993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:53.371120 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.371085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.371321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.371173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz76r\" (UniqueName: \"kubernetes.io/projected/1155c028-9680-4af3-a548-0b4bbca1d763-kube-api-access-mz76r\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.371321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.371218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.371321 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:40:53.371241 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-serving-cert: secret "maas-api-e2e-shared-a-w3-d90e71-serving-cert" not found Sep 11 18:40:53.371321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.371248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1155c028-9680-4af3-a548-0b4bbca1d763-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.371321 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:40:53.371323 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-maas-api-tls podName:1155c028-9680-4af3-a548-0b4bbca1d763 nodeName:}" failed. No retries permitted until 2026-09-11 18:40:53.871301319 +0000 UTC m=+1522.097977652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-maas-api-tls") pod "maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" (UID: "1155c028-9680-4af3-a548-0b4bbca1d763") : secret "maas-api-e2e-shared-a-w3-d90e71-serving-cert" not found Sep 11 18:40:53.372016 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.371992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1155c028-9680-4af3-a548-0b4bbca1d763-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.372159 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.372052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:53.374063 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.374041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.381411 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.381389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz76r\" (UniqueName: \"kubernetes.io/projected/1155c028-9680-4af3-a548-0b4bbca1d763-kube-api-access-mz76r\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.471556 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.471510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:53.802216 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.802184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:53.877287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.877243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.879901 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.879875 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:53.901088 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:53.901054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:54.139912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:54.139809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:54.220247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:54.220201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:54.426503 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:54.426477 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn"] Sep 11 18:40:54.428206 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:40:54.428176 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1155c028_9680_4af3_a548_0b4bbca1d763.slice/crio-7b663c98e5a5e48e08cb1f6e9cfef68ba28f80bef7a324110c2a4eaddb0ffc39 WatchSource:0}: Error finding container 7b663c98e5a5e48e08cb1f6e9cfef68ba28f80bef7a324110c2a4eaddb0ffc39: Status 404 returned error can't find the container with id 7b663c98e5a5e48e08cb1f6e9cfef68ba28f80bef7a324110c2a4eaddb0ffc39 Sep 11 18:40:54.716543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:54.716421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" event={"ID":"1155c028-9680-4af3-a548-0b4bbca1d763","Type":"ContainerStarted","Data":"7b663c98e5a5e48e08cb1f6e9cfef68ba28f80bef7a324110c2a4eaddb0ffc39"} Sep 11 18:40:55.119950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:55.119909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:40:55.723818 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:55.723782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" event={"ID":"1155c028-9680-4af3-a548-0b4bbca1d763","Type":"ContainerStarted","Data":"d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5"} Sep 11 18:40:55.724020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:55.723865 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:40:55.746405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:55.746355 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" podStartSLOduration=2.359850257 podStartE2EDuration="2.746340173s" totalImagesPullingTime="386.489916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:40:55.742814802 +0000 UTC m=+1523.969491162" watchObservedRunningTime="2026-09-11 18:40:55.746340173 +0000 UTC m=+1523.973016512" Sep 11 18:40:57.942691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:40:57.942651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:01.735366 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:01.735338 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:41:03.816863 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:03.816824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:03.902044 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:03.902009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:03.979823 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:03.979788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:04.056557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:04.056496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:04.423131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:04.423099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:04.526302 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:04.526268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:05.448170 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:05.448136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:06.257757 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:06.257723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:07.043816 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:07.043783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:07.284319 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:07.284280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:07.380727 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:07.380622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:07.864751 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:07.864696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:08.358044 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:08.358012 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:41:08.856007 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:08.855968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:09.756773 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:09.756729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:10.735563 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:10.735521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:11.456506 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:11.456472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:11.557539 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:11.557504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:11.658411 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:11.658375 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:11.758491 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:11.758454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:12.538610 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:12.538576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:13.363059 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:13.363028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:13.997609 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:13.997564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:14.296223 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:14.296181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:14.402443 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:14.402403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:14.665674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:14.665574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:14.933845 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:14.933756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:15.076605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:15.076562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:17.528607 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:17.528571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:18.202428 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:18.202388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:18.354781 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:18.354743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:18.845689 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:18.845649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:19.654820 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:19.654774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:20.504555 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:20.504514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:21.244667 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:21.244634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:22.012199 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:22.012164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:22.889875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:22.889839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:23.001159 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:23.001122 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:23.100672 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:23.100640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:23.401533 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:23.401487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:23.496036 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:23.495995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:28.903281 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:28.903246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:28.978441 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:28.978403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:30.258419 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:30.258380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:36.358962 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:36.358923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:36.460509 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:36.460474 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:41:36.684249 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:36.684139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:40.060150 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:40.060118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:42.825610 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:42.825568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:46.063072 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:46.063029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:47.065691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:47.065657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:48.039153 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:48.039118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:48.284085 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:48.284044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:48.931144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:48.931111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:49.972780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:49.972738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:50.781221 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:50.781192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:51.772107 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:51.772073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:52.615485 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:52.615435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:53.301312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:53.301264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:53.971056 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:53.971017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:54.639616 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:54.639579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:55.294645 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:55.294599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:56.212979 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:56.212947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:56.678856 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:56.678822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:41:57.642823 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:57.642787 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:41:59.000549 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:41:59.000505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:00.748133 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:00.748101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:06.310984 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.310942 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc"] Sep 11 18:42:06.311440 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.311205 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" podUID="acba202b-845c-4a2e-bb3b-6f6e009bce05" containerName="maas-api" containerID="cri-o://19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9" gracePeriod=30 Sep 11 18:42:06.566601 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.566541 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:42:06.611095 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.611061 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw9pq\" (UniqueName: \"kubernetes.io/projected/acba202b-845c-4a2e-bb3b-6f6e009bce05-kube-api-access-vw9pq\") pod \"acba202b-845c-4a2e-bb3b-6f6e009bce05\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " Sep 11 18:42:06.611269 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.611137 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-maas-api-tls\") pod \"acba202b-845c-4a2e-bb3b-6f6e009bce05\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " Sep 11 18:42:06.611269 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.611186 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acba202b-845c-4a2e-bb3b-6f6e009bce05-openshift-service-ca\") pod \"acba202b-845c-4a2e-bb3b-6f6e009bce05\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " Sep 11 18:42:06.611269 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.611259 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-metrics-certs\") pod \"acba202b-845c-4a2e-bb3b-6f6e009bce05\" (UID: \"acba202b-845c-4a2e-bb3b-6f6e009bce05\") " Sep 11 18:42:06.611576 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.611547 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acba202b-845c-4a2e-bb3b-6f6e009bce05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "acba202b-845c-4a2e-bb3b-6f6e009bce05" (UID: "acba202b-845c-4a2e-bb3b-6f6e009bce05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:42:06.611725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.611650 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acba202b-845c-4a2e-bb3b-6f6e009bce05-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:42:06.613338 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.613309 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acba202b-845c-4a2e-bb3b-6f6e009bce05-kube-api-access-vw9pq" (OuterVolumeSpecName: "kube-api-access-vw9pq") pod "acba202b-845c-4a2e-bb3b-6f6e009bce05" (UID: "acba202b-845c-4a2e-bb3b-6f6e009bce05"). InnerVolumeSpecName "kube-api-access-vw9pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:42:06.613444 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.613338 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "acba202b-845c-4a2e-bb3b-6f6e009bce05" (UID: "acba202b-845c-4a2e-bb3b-6f6e009bce05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:42:06.613444 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.613350 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "acba202b-845c-4a2e-bb3b-6f6e009bce05" (UID: "acba202b-845c-4a2e-bb3b-6f6e009bce05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:42:06.712504 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.712469 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:42:06.712504 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.712494 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acba202b-845c-4a2e-bb3b-6f6e009bce05-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:42:06.712504 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:06.712503 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9pq\" (UniqueName: \"kubernetes.io/projected/acba202b-845c-4a2e-bb3b-6f6e009bce05-kube-api-access-vw9pq\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.183162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.182855 2579 generic.go:356] "Generic (PLEG): container finished" podID="acba202b-845c-4a2e-bb3b-6f6e009bce05" containerID="19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9" exitCode=0 Sep 11 18:42:07.183162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.182914 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" Sep 11 18:42:07.183162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.182914 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" event={"ID":"acba202b-845c-4a2e-bb3b-6f6e009bce05","Type":"ContainerDied","Data":"19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9"} Sep 11 18:42:07.183162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.182985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc" event={"ID":"acba202b-845c-4a2e-bb3b-6f6e009bce05","Type":"ContainerDied","Data":"eb8b23cdb0b1e2e4fbeb9a1ea2feccfdf23515c71adac636074359652733b5c1"} Sep 11 18:42:07.183162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.183017 2579 scope.go:117] "RemoveContainer" containerID="19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9" Sep 11 18:42:07.187097 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.187071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:07.194576 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.194561 2579 scope.go:117] "RemoveContainer" containerID="19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9" Sep 11 18:42:07.194857 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:42:07.194839 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9\": container with ID starting with 19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9 not found: ID does not exist" containerID="19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9" Sep 11 18:42:07.194910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.194863 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9"} err="failed to get container status \"19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9\": rpc error: code = NotFound desc = could not find container \"19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9\": container with ID starting with 19033f252f40ec1ec4fb06414fe7ecc1b2fe71d7e7b3f5a8e976b11ce61a94a9 not found: ID does not exist" Sep 11 18:42:07.209612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.209587 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc"] Sep 11 18:42:07.219081 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.219060 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-699caf96-577cdf579-9xptc"] Sep 11 18:42:07.284873 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.284829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:07.744955 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:07.744918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:08.230009 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:08.229962 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acba202b-845c-4a2e-bb3b-6f6e009bce05" path="/var/lib/kubelet/pods/acba202b-845c-4a2e-bb3b-6f6e009bce05/volumes" Sep 11 18:42:08.534697 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:08.534658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:09.233838 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:09.233799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:09.741775 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:09.741728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:09.963643 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:09.963608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:10.643386 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:10.643348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:14.780839 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:14.780802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:16.438528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:16.438492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:22.546338 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:22.546300 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:42:27.519632 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:27.519554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:51.990263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:51.990222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:52.132255 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:52.132218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:57.659025 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:57.658989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:42:59.319574 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:42:59.319540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:06.967168 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:06.967130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:10.138037 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:10.137999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:10.244591 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:10.244506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:10.485777 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:10.485737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:12.461815 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:12.461779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:12.562752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:12.562719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:12.685266 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:12.685233 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7"] Sep 11 18:43:12.685518 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:12.685488 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" containerID="cri-o://16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da" gracePeriod=30 Sep 11 18:43:12.702894 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:12.702832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:13.957562 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:13.957525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:14.056945 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:14.056910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:14.158265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:14.158228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:14.770171 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:14.770138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:14.908165 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:14.908126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:15.595981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:15.595939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:16.267360 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:16.267324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:17.947883 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:17.947816 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:43:18.039780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.039751 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-certs\") pod \"d05d5277-d4b5-46b4-a216-472e13a7c580\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " Sep 11 18:43:18.039780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.039787 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-data\") pod \"d05d5277-d4b5-46b4-a216-472e13a7c580\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " Sep 11 18:43:18.039988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.039809 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-token\") pod \"d05d5277-d4b5-46b4-a216-472e13a7c580\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " Sep 11 18:43:18.039988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.039851 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-podinfo\") pod \"d05d5277-d4b5-46b4-a216-472e13a7c580\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " Sep 11 18:43:18.039988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.039872 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-socket\") pod \"d05d5277-d4b5-46b4-a216-472e13a7c580\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " Sep 11 18:43:18.039988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.039892 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-credential-socket\") pod \"d05d5277-d4b5-46b4-a216-472e13a7c580\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " Sep 11 18:43:18.039988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.039981 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d5277-d4b5-46b4-a216-472e13a7c580-istiod-ca-cert\") pod \"d05d5277-d4b5-46b4-a216-472e13a7c580\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " Sep 11 18:43:18.040405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040014 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwgvd\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-kube-api-access-zwgvd\") pod \"d05d5277-d4b5-46b4-a216-472e13a7c580\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " Sep 11 18:43:18.040405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040078 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-envoy\") pod \"d05d5277-d4b5-46b4-a216-472e13a7c580\" (UID: \"d05d5277-d4b5-46b4-a216-472e13a7c580\") " Sep 11 18:43:18.040405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040086 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-data" (OuterVolumeSpecName: "istio-data") pod "d05d5277-d4b5-46b4-a216-472e13a7c580" (UID: "d05d5277-d4b5-46b4-a216-472e13a7c580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:43:18.040405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040098 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d05d5277-d4b5-46b4-a216-472e13a7c580" (UID: "d05d5277-d4b5-46b4-a216-472e13a7c580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:43:18.040405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040150 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d05d5277-d4b5-46b4-a216-472e13a7c580" (UID: "d05d5277-d4b5-46b4-a216-472e13a7c580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:43:18.040405 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040269 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d05d5277-d4b5-46b4-a216-472e13a7c580" (UID: "d05d5277-d4b5-46b4-a216-472e13a7c580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:43:18.040783 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040431 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:43:18.040783 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040434 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05d5277-d4b5-46b4-a216-472e13a7c580-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d05d5277-d4b5-46b4-a216-472e13a7c580" (UID: "d05d5277-d4b5-46b4-a216-472e13a7c580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:43:18.040783 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040452 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-credential-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:43:18.040783 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040464 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-workload-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:43:18.040783 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.040477 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-data\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:43:18.042166 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.042144 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-token" (OuterVolumeSpecName: "istio-token") pod "d05d5277-d4b5-46b4-a216-472e13a7c580" (UID: "d05d5277-d4b5-46b4-a216-472e13a7c580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:43:18.042295 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.042271 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d05d5277-d4b5-46b4-a216-472e13a7c580" (UID: "d05d5277-d4b5-46b4-a216-472e13a7c580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:43:18.042492 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.042470 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d05d5277-d4b5-46b4-a216-472e13a7c580" (UID: "d05d5277-d4b5-46b4-a216-472e13a7c580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:43:18.042562 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.042544 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-kube-api-access-zwgvd" (OuterVolumeSpecName: "kube-api-access-zwgvd") pod "d05d5277-d4b5-46b4-a216-472e13a7c580" (UID: "d05d5277-d4b5-46b4-a216-472e13a7c580"). InnerVolumeSpecName "kube-api-access-zwgvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:43:18.141962 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.141928 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-envoy\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:43:18.141962 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.141955 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:43:18.141962 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.141965 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d5277-d4b5-46b4-a216-472e13a7c580-istio-podinfo\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:43:18.142207 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.141973 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d5277-d4b5-46b4-a216-472e13a7c580-istiod-ca-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:43:18.142207 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.141984 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwgvd\" (UniqueName: \"kubernetes.io/projected/d05d5277-d4b5-46b4-a216-472e13a7c580-kube-api-access-zwgvd\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:43:18.621690 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.621650 2579 generic.go:356] "Generic (PLEG): container finished" podID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerID="16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da" exitCode=0 Sep 11 18:43:18.621875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.621699 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" Sep 11 18:43:18.621875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.621739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" event={"ID":"d05d5277-d4b5-46b4-a216-472e13a7c580","Type":"ContainerDied","Data":"16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da"} Sep 11 18:43:18.621875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.621805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7" event={"ID":"d05d5277-d4b5-46b4-a216-472e13a7c580","Type":"ContainerDied","Data":"8bf4c9e2fc409411853c4342e17ee88b1e8fa6dd332468d926beae7c766adf59"} Sep 11 18:43:18.621875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.621837 2579 scope.go:117] "RemoveContainer" containerID="16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da" Sep 11 18:43:18.631568 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.631552 2579 scope.go:117] "RemoveContainer" containerID="de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047" Sep 11 18:43:18.641359 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.641340 2579 scope.go:117] "RemoveContainer" containerID="16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da" Sep 11 18:43:18.641682 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:43:18.641657 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da\": container with ID starting with 16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da not found: ID does not exist" containerID="16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da" Sep 11 18:43:18.641888 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.641723 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da"} err="failed to get container status \"16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da\": rpc error: code = NotFound desc = could not find container \"16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da\": container with ID starting with 16d10797989a21066003a0931922cdb7a33c8b5358ed40db7becee65a44206da not found: ID does not exist" Sep 11 18:43:18.641888 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.641747 2579 scope.go:117] "RemoveContainer" containerID="de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047" Sep 11 18:43:18.642137 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:43:18.642115 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047\": container with ID starting with de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047 not found: ID does not exist" containerID="de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047" Sep 11 18:43:18.642197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.642146 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047"} err="failed to get container status \"de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047\": rpc error: code = NotFound desc = could not find container \"de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047\": container with ID starting with de8f4f8b6665f1a474d1d36dfcf36ae3caef750152dabda524a2f79af967f047 not found: ID does not exist" Sep 11 18:43:18.642645 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.642463 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7"] Sep 11 18:43:18.645256 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:18.645236 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-699caf96-openshift-default-847c959b56-mfqr7"] Sep 11 18:43:20.235182 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:20.235148 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" path="/var/lib/kubelet/pods/d05d5277-d4b5-46b4-a216-472e13a7c580/volumes" Sep 11 18:43:20.944842 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:20.944813 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:43:22.235757 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:22.235698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:23.613164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:23.613127 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:26.395076 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:26.395047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:27.030032 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:27.029988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:29.694107 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:29.694062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:29.798910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:29.798854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:29.895599 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:29.895555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:39.147203 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:39.147161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:40.405148 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:40.405117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:40.654157 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:40.654120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:45.353897 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:45.353863 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:43:53.114105 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:53.114050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:43:53.179679 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:43:53.179632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:01.562671 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:01.562578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:07.251407 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:07.251375 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:07.351428 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:07.351377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:07.581276 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:07.581177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:07.889263 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:07.889173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:08.328357 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:08.328325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:09.846832 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:09.846794 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:44:10.537942 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:10.537911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:11.333158 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:11.333106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:11.431281 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:11.431241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:12.380177 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:12.380139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:13.141536 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:13.141496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:14.005629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:14.005589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:14.702817 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:14.702779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:15.332007 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.331968 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9"] Sep 11 18:44:15.332565 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332545 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acba202b-845c-4a2e-bb3b-6f6e009bce05" containerName="maas-api" Sep 11 18:44:15.332658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332567 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="acba202b-845c-4a2e-bb3b-6f6e009bce05" containerName="maas-api" Sep 11 18:44:15.332658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332584 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" Sep 11 18:44:15.332658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332593 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" Sep 11 18:44:15.332658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332603 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" Sep 11 18:44:15.332658 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332611 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" Sep 11 18:44:15.332951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332762 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" Sep 11 18:44:15.332951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332775 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f015fd-0119-494c-96bc-8726b34166ad" containerName="istio-proxy" Sep 11 18:44:15.332951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332785 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="acba202b-845c-4a2e-bb3b-6f6e009bce05" containerName="maas-api" Sep 11 18:44:15.332951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.332796 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" Sep 11 18:44:15.336473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.336452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.338908 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.338884 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-883e85-serving-cert\"" Sep 11 18:44:15.348464 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.348436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9"] Sep 11 18:44:15.368899 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.368863 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:15.427381 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.427343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:15.512743 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.512681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-metrics-certs\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.512911 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.512818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz6qb\" (UniqueName: \"kubernetes.io/projected/17297b36-887b-4578-9c36-6c9f3996ba94-kube-api-access-hz6qb\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.512957 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.512922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17297b36-887b-4578-9c36-6c9f3996ba94-openshift-service-ca\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.512995 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.512976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-maas-api-tls\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.569842 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.569806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:15.613981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.613892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-maas-api-tls\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.613981 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.613952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-metrics-certs\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.614229 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.614009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz6qb\" (UniqueName: \"kubernetes.io/projected/17297b36-887b-4578-9c36-6c9f3996ba94-kube-api-access-hz6qb\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.614229 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.614067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17297b36-887b-4578-9c36-6c9f3996ba94-openshift-service-ca\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.614657 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.614637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17297b36-887b-4578-9c36-6c9f3996ba94-openshift-service-ca\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.616499 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.616471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-maas-api-tls\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.616612 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.616539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-metrics-certs\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.622004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.621982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz6qb\" (UniqueName: \"kubernetes.io/projected/17297b36-887b-4578-9c36-6c9f3996ba94-kube-api-access-hz6qb\") pod \"maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.648077 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.648036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:15.942760 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.942647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:15.951559 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.951528 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9"] Sep 11 18:44:15.953001 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:44:15.952972 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17297b36_887b_4578_9c36_6c9f3996ba94.slice/crio-09ecb51852b9f67e1ed5bef76a75c98ba90536c22e2e55cef871ed67078dba4f WatchSource:0}: Error finding container 09ecb51852b9f67e1ed5bef76a75c98ba90536c22e2e55cef871ed67078dba4f: Status 404 returned error can't find the container with id 09ecb51852b9f67e1ed5bef76a75c98ba90536c22e2e55cef871ed67078dba4f Sep 11 18:44:15.954410 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.954391 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:44:15.981781 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:15.981742 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" event={"ID":"17297b36-887b-4578-9c36-6c9f3996ba94","Type":"ContainerStarted","Data":"09ecb51852b9f67e1ed5bef76a75c98ba90536c22e2e55cef871ed67078dba4f"} Sep 11 18:44:16.092766 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:16.092732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:16.992222 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:16.992185 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" event={"ID":"17297b36-887b-4578-9c36-6c9f3996ba94","Type":"ContainerStarted","Data":"f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528"} Sep 11 18:44:16.992783 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:16.992299 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:17.016781 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:17.016731 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" podStartSLOduration=1.565544426 podStartE2EDuration="2.016715734s" totalImagesPullingTime="451.171308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:17.013338668 +0000 UTC m=+1725.240015029" watchObservedRunningTime="2026-09-11 18:44:17.016715734 +0000 UTC m=+1725.243392067" Sep 11 18:44:23.004753 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:23.004723 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:44:37.343441 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:37.343401 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:44:37.521487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:37.521455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:38.316851 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:38.316809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:39.192651 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:39.192613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:39.910688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:39.910643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:40.774787 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:40.774754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:41.268674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:41.268637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:41.502010 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:41.501978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:41.706073 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:41.705981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:42.027655 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:42.027618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:42.243753 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:42.243681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:42.349678 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:42.349549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:44.675512 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:44.675477 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:45.422991 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:45.422947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:45.522135 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:45.522095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:46.303486 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:46.303443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:47.234654 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:47.234599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:48.019585 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:48.019545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:48.830599 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:48.830557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:49.616798 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:49.616755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:50.399278 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.399239 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp"] Sep 11 18:44:50.400023 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.399996 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" Sep 11 18:44:50.400023 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.400024 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05d5277-d4b5-46b4-a216-472e13a7c580" containerName="istio-proxy" Sep 11 18:44:50.404104 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.404082 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.407663 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.407638 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5cbc66-serving-cert\"" Sep 11 18:44:50.408960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.408939 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp"] Sep 11 18:44:50.432867 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.432832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:50.451179 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.451154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-maas-api-tls\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.451296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.451199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kz6\" (UniqueName: \"kubernetes.io/projected/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-kube-api-access-r4kz6\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.451296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.451285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-openshift-service-ca\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.451399 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.451384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-metrics-certs\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.534664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.534628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:50.552838 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.552800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-metrics-certs\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.553021 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.552860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-maas-api-tls\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.553021 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.552892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kz6\" (UniqueName: \"kubernetes.io/projected/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-kube-api-access-r4kz6\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.553021 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.552917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-openshift-service-ca\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.553021 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:44:50.552972 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-serving-cert: secret "maas-api-e2e-api-b-5cbc66-serving-cert" not found Sep 11 18:44:50.553188 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:44:50.553039 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-maas-api-tls podName:a98b0651-9f57-44b3-871a-89c2ff8c9f4e nodeName:}" failed. No retries permitted until 2026-09-11 18:44:51.053016563 +0000 UTC m=+1759.279692894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-maas-api-tls") pod "maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" (UID: "a98b0651-9f57-44b3-871a-89c2ff8c9f4e") : secret "maas-api-e2e-api-b-5cbc66-serving-cert" not found Sep 11 18:44:50.553534 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.553516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-openshift-service-ca\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.555393 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.555370 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-metrics-certs\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.563431 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.563405 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kz6\" (UniqueName: \"kubernetes.io/projected/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-kube-api-access-r4kz6\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:50.693105 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.692994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:50.902133 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:50.902096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:51.003071 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:51.003037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:44:51.057424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:51.057388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-maas-api-tls\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:51.059861 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:51.059837 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-maas-api-tls\") pod \"maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:51.316811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:51.316690 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:51.591820 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:51.591789 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp"] Sep 11 18:44:51.593055 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:44:51.593024 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98b0651_9f57_44b3_871a_89c2ff8c9f4e.slice/crio-93206a1149305191a6edd6fed7640a1a605125d29fe99bc7fc60c8c3a99b5429 WatchSource:0}: Error finding container 93206a1149305191a6edd6fed7640a1a605125d29fe99bc7fc60c8c3a99b5429: Status 404 returned error can't find the container with id 93206a1149305191a6edd6fed7640a1a605125d29fe99bc7fc60c8c3a99b5429 Sep 11 18:44:52.209108 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:52.209074 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" event={"ID":"a98b0651-9f57-44b3-871a-89c2ff8c9f4e","Type":"ContainerStarted","Data":"93206a1149305191a6edd6fed7640a1a605125d29fe99bc7fc60c8c3a99b5429"} Sep 11 18:44:53.217691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:53.217650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" event={"ID":"a98b0651-9f57-44b3-871a-89c2ff8c9f4e","Type":"ContainerStarted","Data":"c695c07590ebb9a48e93cdd44398d69ee45e441f03ffa47de8edff5604552f01"} Sep 11 18:44:53.218123 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:53.217736 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:44:53.238651 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:53.238599 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" podStartSLOduration=2.557486861 podStartE2EDuration="3.238583699s" totalImagesPullingTime="681.096838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:53.235404766 +0000 UTC m=+1761.462081126" watchObservedRunningTime="2026-09-11 18:44:53.238583699 +0000 UTC m=+1761.465260031" Sep 11 18:44:59.231246 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:44:59.231217 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:45:03.540557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:03.540514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:05.554420 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:05.554384 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:45:12.506763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:12.506726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:13.215898 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:13.215865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:14.015022 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:14.014964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:14.675457 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:14.675416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:15.335272 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:15.335230 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:15.434615 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:15.434579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:16.052960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:16.052923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:16.716627 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:16.716580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:17.636608 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:17.636565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:17.724998 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:17.724962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:17.825064 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:17.825027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:18.394068 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:18.394029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:19.221869 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:19.221834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:19.918120 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:19.918088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:20.743923 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:20.743885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:21.506625 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:21.506574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:22.181054 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:22.181021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:22.847064 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:22.847022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:23.520218 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:23.520174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:24.319479 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:24.319448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:32.394099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:32.394069 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:45:32.400950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:32.400928 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:45:33.544503 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:33.544459 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:45:33.775603 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:33.775561 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9"] Sep 11 18:45:33.775872 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:33.775846 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" podUID="17297b36-887b-4578-9c36-6c9f3996ba94" containerName="maas-api" containerID="cri-o://f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528" gracePeriod=30 Sep 11 18:45:33.819241 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:33.819210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:33.919319 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:33.919224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:34.005618 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.005574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:34.030763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.030740 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:45:34.074336 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.074252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:34.205484 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.205456 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-metrics-certs\") pod \"17297b36-887b-4578-9c36-6c9f3996ba94\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " Sep 11 18:45:34.205679 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.205495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz6qb\" (UniqueName: \"kubernetes.io/projected/17297b36-887b-4578-9c36-6c9f3996ba94-kube-api-access-hz6qb\") pod \"17297b36-887b-4578-9c36-6c9f3996ba94\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " Sep 11 18:45:34.205679 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.205531 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17297b36-887b-4578-9c36-6c9f3996ba94-openshift-service-ca\") pod \"17297b36-887b-4578-9c36-6c9f3996ba94\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " Sep 11 18:45:34.205679 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.205601 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-maas-api-tls\") pod \"17297b36-887b-4578-9c36-6c9f3996ba94\" (UID: \"17297b36-887b-4578-9c36-6c9f3996ba94\") " Sep 11 18:45:34.205985 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.205955 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17297b36-887b-4578-9c36-6c9f3996ba94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17297b36-887b-4578-9c36-6c9f3996ba94" (UID: "17297b36-887b-4578-9c36-6c9f3996ba94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:45:34.207715 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.207676 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17297b36-887b-4578-9c36-6c9f3996ba94" (UID: "17297b36-887b-4578-9c36-6c9f3996ba94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:45:34.207715 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.207693 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "17297b36-887b-4578-9c36-6c9f3996ba94" (UID: "17297b36-887b-4578-9c36-6c9f3996ba94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:45:34.207860 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.207746 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17297b36-887b-4578-9c36-6c9f3996ba94-kube-api-access-hz6qb" (OuterVolumeSpecName: "kube-api-access-hz6qb") pod "17297b36-887b-4578-9c36-6c9f3996ba94" (UID: "17297b36-887b-4578-9c36-6c9f3996ba94"). InnerVolumeSpecName "kube-api-access-hz6qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:45:34.306694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.306656 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:45:34.306694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.306683 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17297b36-887b-4578-9c36-6c9f3996ba94-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:45:34.306694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.306694 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz6qb\" (UniqueName: \"kubernetes.io/projected/17297b36-887b-4578-9c36-6c9f3996ba94-kube-api-access-hz6qb\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:45:34.306694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.306728 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17297b36-887b-4578-9c36-6c9f3996ba94-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:45:34.359763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.359640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:34.460747 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.460698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:34.477024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.476989 2579 generic.go:356] "Generic (PLEG): container finished" podID="17297b36-887b-4578-9c36-6c9f3996ba94" containerID="f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528" exitCode=0 Sep 11 18:45:34.477204 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.477032 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" Sep 11 18:45:34.477204 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.477072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" event={"ID":"17297b36-887b-4578-9c36-6c9f3996ba94","Type":"ContainerDied","Data":"f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528"} Sep 11 18:45:34.477204 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.477124 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9" event={"ID":"17297b36-887b-4578-9c36-6c9f3996ba94","Type":"ContainerDied","Data":"09ecb51852b9f67e1ed5bef76a75c98ba90536c22e2e55cef871ed67078dba4f"} Sep 11 18:45:34.477204 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.477147 2579 scope.go:117] "RemoveContainer" containerID="f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528" Sep 11 18:45:34.488677 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.488655 2579 scope.go:117] "RemoveContainer" containerID="f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528" Sep 11 18:45:34.489013 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:45:34.488991 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528\": container with ID starting with f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528 not found: ID does not exist" containerID="f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528" Sep 11 18:45:34.489111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.489023 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528"} err="failed to get container status \"f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528\": rpc error: code = NotFound desc = could not find container \"f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528\": container with ID starting with f65ff4e280c50f22394bdb195bda5a91358990ba34e8a072f9e1c9e9e99d4528 not found: ID does not exist" Sep 11 18:45:34.501496 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.501470 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9"] Sep 11 18:45:34.510990 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:34.510963 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-883e85-6d7fdc8746-c6gn9"] Sep 11 18:45:35.116496 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:35.116439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:36.018848 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:36.018816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:36.230296 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:36.230262 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17297b36-887b-4578-9c36-6c9f3996ba94" path="/var/lib/kubelet/pods/17297b36-887b-4578-9c36-6c9f3996ba94/volumes" Sep 11 18:45:36.710629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:36.710596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:37.508945 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:37.508910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:38.207969 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:38.207939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:39.011960 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:39.011926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:40.242783 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:40.242750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:40.343277 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:40.343233 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:40.517905 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:40.517871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:40.933172 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:40.933074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:41.627014 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:41.626979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:42.394928 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:42.394894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:43.055153 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:43.055108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:43.708507 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:43.708463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:44.535644 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:44.535609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:45.201005 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:45.200973 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:45.881606 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:45.881569 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:56.430388 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.430201 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp"] Sep 11 18:45:56.430890 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.430472 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" podUID="a98b0651-9f57-44b3-871a-89c2ff8c9f4e" containerName="maas-api" containerID="cri-o://c695c07590ebb9a48e93cdd44398d69ee45e441f03ffa47de8edff5604552f01" gracePeriod=30 Sep 11 18:45:56.462411 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.462379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:56.564062 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.564018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:56.621856 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.621821 2579 generic.go:356] "Generic (PLEG): container finished" podID="a98b0651-9f57-44b3-871a-89c2ff8c9f4e" containerID="c695c07590ebb9a48e93cdd44398d69ee45e441f03ffa47de8edff5604552f01" exitCode=0 Sep 11 18:45:56.622030 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.621886 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" event={"ID":"a98b0651-9f57-44b3-871a-89c2ff8c9f4e","Type":"ContainerDied","Data":"c695c07590ebb9a48e93cdd44398d69ee45e441f03ffa47de8edff5604552f01"} Sep 11 18:45:56.661667 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.661631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:56.685160 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.685094 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:45:56.723916 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.723888 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-maas-api-tls\") pod \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " Sep 11 18:45:56.724099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.723968 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-openshift-service-ca\") pod \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " Sep 11 18:45:56.724099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.723987 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-metrics-certs\") pod \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " Sep 11 18:45:56.724099 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.724031 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4kz6\" (UniqueName: \"kubernetes.io/projected/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-kube-api-access-r4kz6\") pod \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\" (UID: \"a98b0651-9f57-44b3-871a-89c2ff8c9f4e\") " Sep 11 18:45:56.724460 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.724425 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a98b0651-9f57-44b3-871a-89c2ff8c9f4e" (UID: "a98b0651-9f57-44b3-871a-89c2ff8c9f4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:45:56.726436 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.726396 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-kube-api-access-r4kz6" (OuterVolumeSpecName: "kube-api-access-r4kz6") pod "a98b0651-9f57-44b3-871a-89c2ff8c9f4e" (UID: "a98b0651-9f57-44b3-871a-89c2ff8c9f4e"). InnerVolumeSpecName "kube-api-access-r4kz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:45:56.726759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.726734 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a98b0651-9f57-44b3-871a-89c2ff8c9f4e" (UID: "a98b0651-9f57-44b3-871a-89c2ff8c9f4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:45:56.726759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.726747 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a98b0651-9f57-44b3-871a-89c2ff8c9f4e" (UID: "a98b0651-9f57-44b3-871a-89c2ff8c9f4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:45:56.763619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.763567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:56.826067 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.826021 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:45:56.826067 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.826058 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:45:56.826067 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.826074 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:45:56.826378 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:56.826087 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kz6\" (UniqueName: \"kubernetes.io/projected/a98b0651-9f57-44b3-871a-89c2ff8c9f4e-kube-api-access-r4kz6\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:45:57.025794 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:57.025760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:57.151602 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:57.151568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:57.629197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:57.629154 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" event={"ID":"a98b0651-9f57-44b3-871a-89c2ff8c9f4e","Type":"ContainerDied","Data":"93206a1149305191a6edd6fed7640a1a605125d29fe99bc7fc60c8c3a99b5429"} Sep 11 18:45:57.629197 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:57.629180 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp" Sep 11 18:45:57.629678 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:57.629226 2579 scope.go:117] "RemoveContainer" containerID="c695c07590ebb9a48e93cdd44398d69ee45e441f03ffa47de8edff5604552f01" Sep 11 18:45:57.651304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:57.651275 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp"] Sep 11 18:45:57.655306 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:57.655275 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5cbc66-ff6767675-vnrtp"] Sep 11 18:45:58.039691 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:58.039654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:58.229997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:58.229965 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98b0651-9f57-44b3-871a-89c2ff8c9f4e" path="/var/lib/kubelet/pods/a98b0651-9f57-44b3-871a-89c2ff8c9f4e/volumes" Sep 11 18:45:58.743231 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:58.743194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:45:59.344171 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:59.344137 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:45:59.379953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:45:59.379917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:00.040389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:00.040345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:00.695301 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:00.695262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:01.482941 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:01.482875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:01.686171 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:01.686130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:02.977473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:02.977433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:03.078684 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:03.078650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:03.235570 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:03.235473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:03.910521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:03.910480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:04.649526 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:04.649486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:05.418662 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:05.418631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:06.073577 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:06.073541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:06.755392 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:06.755359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:07.477778 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:07.477738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:08.201862 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:08.201819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.013950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.013917 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz"] Sep 11 18:46:14.014793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.014730 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" podUID="be80f334-e76f-45b8-b3ba-14994c4e9d67" containerName="maas-api" containerID="cri-o://82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d" gracePeriod=30 Sep 11 18:46:14.039304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.039260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.107470 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.107431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.147269 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.147232 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" podUID="be80f334-e76f-45b8-b3ba-14994c4e9d67" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.60:8443/health\": dial tcp 10.132.0.60:8443: connect: connection refused" Sep 11 18:46:14.169421 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.169386 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z"] Sep 11 18:46:14.169737 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.169616 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" podUID="d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" containerName="maas-api" containerID="cri-o://5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440" gracePeriod=30 Sep 11 18:46:14.178108 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.177785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.260614 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.260563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.326548 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.326484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.399254 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.399221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.445573 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.445541 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:46:14.458926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.458883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.498228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.498202 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:46:14.604521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.604424 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-openshift-service-ca\") pod \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " Sep 11 18:46:14.604521 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.604511 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tg96\" (UniqueName: \"kubernetes.io/projected/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-kube-api-access-2tg96\") pod \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " Sep 11 18:46:14.604805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.604588 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be80f334-e76f-45b8-b3ba-14994c4e9d67-openshift-service-ca\") pod \"be80f334-e76f-45b8-b3ba-14994c4e9d67\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " Sep 11 18:46:14.604805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.604624 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-metrics-certs\") pod \"be80f334-e76f-45b8-b3ba-14994c4e9d67\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " Sep 11 18:46:14.604805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.604648 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-maas-api-tls\") pod \"be80f334-e76f-45b8-b3ba-14994c4e9d67\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " Sep 11 18:46:14.604805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.604669 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghjp5\" (UniqueName: \"kubernetes.io/projected/be80f334-e76f-45b8-b3ba-14994c4e9d67-kube-api-access-ghjp5\") pod \"be80f334-e76f-45b8-b3ba-14994c4e9d67\" (UID: \"be80f334-e76f-45b8-b3ba-14994c4e9d67\") " Sep 11 18:46:14.604805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.604698 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-metrics-certs\") pod \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " Sep 11 18:46:14.605130 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.605105 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-maas-api-tls\") pod \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\" (UID: \"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a\") " Sep 11 18:46:14.605265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.604909 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" (UID: "d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:46:14.605265 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.604964 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be80f334-e76f-45b8-b3ba-14994c4e9d67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be80f334-e76f-45b8-b3ba-14994c4e9d67" (UID: "be80f334-e76f-45b8-b3ba-14994c4e9d67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:46:14.605550 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.605526 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be80f334-e76f-45b8-b3ba-14994c4e9d67-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:14.605647 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.605554 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:14.606987 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.606958 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be80f334-e76f-45b8-b3ba-14994c4e9d67" (UID: "be80f334-e76f-45b8-b3ba-14994c4e9d67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:46:14.607094 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.606960 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" (UID: "d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:46:14.607349 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.607325 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be80f334-e76f-45b8-b3ba-14994c4e9d67-kube-api-access-ghjp5" (OuterVolumeSpecName: "kube-api-access-ghjp5") pod "be80f334-e76f-45b8-b3ba-14994c4e9d67" (UID: "be80f334-e76f-45b8-b3ba-14994c4e9d67"). InnerVolumeSpecName "kube-api-access-ghjp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:46:14.607349 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.607342 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-kube-api-access-2tg96" (OuterVolumeSpecName: "kube-api-access-2tg96") pod "d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" (UID: "d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a"). InnerVolumeSpecName "kube-api-access-2tg96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:46:14.607497 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.607326 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "be80f334-e76f-45b8-b3ba-14994c4e9d67" (UID: "be80f334-e76f-45b8-b3ba-14994c4e9d67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:46:14.607728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.607693 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" (UID: "d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:46:14.608527 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.608493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.707116 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.707068 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tg96\" (UniqueName: \"kubernetes.io/projected/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-kube-api-access-2tg96\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:14.707116 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.707100 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:14.707116 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.707111 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be80f334-e76f-45b8-b3ba-14994c4e9d67-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:14.707116 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.707120 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghjp5\" (UniqueName: \"kubernetes.io/projected/be80f334-e76f-45b8-b3ba-14994c4e9d67-kube-api-access-ghjp5\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:14.707116 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.707129 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:14.707544 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.707138 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:14.735801 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.735760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:14.742300 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.742267 2579 generic.go:356] "Generic (PLEG): container finished" podID="be80f334-e76f-45b8-b3ba-14994c4e9d67" containerID="82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d" exitCode=0 Sep 11 18:46:14.742475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.742313 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" Sep 11 18:46:14.742475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.742358 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" event={"ID":"be80f334-e76f-45b8-b3ba-14994c4e9d67","Type":"ContainerDied","Data":"82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d"} Sep 11 18:46:14.742475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.742410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz" event={"ID":"be80f334-e76f-45b8-b3ba-14994c4e9d67","Type":"ContainerDied","Data":"87e444acf0c9b190dc16fda5edc083e741e18d533f1ec25c33ccaae542676b08"} Sep 11 18:46:14.742475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.742439 2579 scope.go:117] "RemoveContainer" containerID="82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d" Sep 11 18:46:14.743995 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.743971 2579 generic.go:356] "Generic (PLEG): container finished" podID="d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" containerID="5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440" exitCode=0 Sep 11 18:46:14.744109 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.744025 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" Sep 11 18:46:14.744109 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.744036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" event={"ID":"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a","Type":"ContainerDied","Data":"5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440"} Sep 11 18:46:14.744109 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.744089 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z" event={"ID":"d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a","Type":"ContainerDied","Data":"335745e38784ec41c54735396023a52ddacd785ce86ceb8c8cc4f5164dae0112"} Sep 11 18:46:14.754656 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.754519 2579 scope.go:117] "RemoveContainer" containerID="82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d" Sep 11 18:46:14.755054 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:46:14.755019 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d\": container with ID starting with 82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d not found: ID does not exist" containerID="82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d" Sep 11 18:46:14.755202 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.755110 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d"} err="failed to get container status \"82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d\": rpc error: code = NotFound desc = could not find container \"82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d\": container with ID starting with 82849898b83eabb1e7c61ecce2a237e1824e9abff74c082cd40774fe34f7cf7d not found: ID does not exist" Sep 11 18:46:14.755202 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.755163 2579 scope.go:117] "RemoveContainer" containerID="5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440" Sep 11 18:46:14.765965 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.765943 2579 scope.go:117] "RemoveContainer" containerID="5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440" Sep 11 18:46:14.766352 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:46:14.766331 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440\": container with ID starting with 5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440 not found: ID does not exist" containerID="5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440" Sep 11 18:46:14.766467 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.766360 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440"} err="failed to get container status \"5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440\": rpc error: code = NotFound desc = could not find container \"5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440\": container with ID starting with 5071ccc9685e741e307e4944e353f439d8a944d457130a885bd6c59f31c36440 not found: ID does not exist" Sep 11 18:46:14.772871 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.772844 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz"] Sep 11 18:46:14.778308 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.778275 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-200c07-59f5b9668f-tkgjz"] Sep 11 18:46:14.792839 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.792810 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z"] Sep 11 18:46:14.793279 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.793256 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72c98d-57984cd4cc-5ss8z"] Sep 11 18:46:14.915093 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:14.914993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:15.625878 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:15.625842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:16.229418 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:16.229381 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be80f334-e76f-45b8-b3ba-14994c4e9d67" path="/var/lib/kubelet/pods/be80f334-e76f-45b8-b3ba-14994c4e9d67/volumes" Sep 11 18:46:16.229855 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:16.229840 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" path="/var/lib/kubelet/pods/d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a/volumes" Sep 11 18:46:16.303801 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:16.303761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:17.033517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:17.033472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:17.830327 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:17.830296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:18.613633 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:18.613601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:20.829616 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:20.829580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:20.930816 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:20.930779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:21.082169 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:21.082083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:21.416279 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:21.416187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:21.475384 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:21.475348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:21.654690 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:21.654656 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q"] Sep 11 18:46:21.654911 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:21.654887 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" containerID="cri-o://3706159a9fdd3f1b2229daab403351e7028e483d31260769505e805fbb2beb5c" gracePeriod=30 Sep 11 18:46:21.684775 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:21.671097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:22.309275 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:22.309239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:23.099975 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:23.099935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:23.819426 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:23.819370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:24.489636 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:24.489599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:24.679592 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:24.679556 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:46:24.679802 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:24.679619 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:46:25.208796 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:25.208761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:26.833421 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:26.833390 2579 generic.go:356] "Generic (PLEG): container finished" podID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerID="3706159a9fdd3f1b2229daab403351e7028e483d31260769505e805fbb2beb5c" exitCode=0 Sep 11 18:46:26.833853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:26.833430 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" event={"ID":"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7","Type":"ContainerDied","Data":"3706159a9fdd3f1b2229daab403351e7028e483d31260769505e805fbb2beb5c"} Sep 11 18:46:26.833853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:26.833485 2579 scope.go:117] "RemoveContainer" containerID="1c78486fd2c4d589155745c8328c28833433146edeb95d84ac5b1278ea1dffd8" Sep 11 18:46:26.918759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:26.918662 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:46:27.021507 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021465 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-credential-socket\") pod \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " Sep 11 18:46:27.021731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021528 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-certs\") pod \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " Sep 11 18:46:27.021731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021560 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-envoy\") pod \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " Sep 11 18:46:27.021731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021584 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-token\") pod \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " Sep 11 18:46:27.021731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021623 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-podinfo\") pod \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " Sep 11 18:46:27.021731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021650 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hblkr\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-kube-api-access-hblkr\") pod \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " Sep 11 18:46:27.021731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021728 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istiod-ca-cert\") pod \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " Sep 11 18:46:27.022061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021754 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-data\") pod \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " Sep 11 18:46:27.022061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021779 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-socket\") pod \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\" (UID: \"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7\") " Sep 11 18:46:27.022061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021842 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" (UID: "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:27.022061 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.021886 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" (UID: "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:27.022247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.022056 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" (UID: "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:27.022247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.022101 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" (UID: "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:27.022247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.022138 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" (UID: "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:46:27.022247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.022206 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-data\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:27.022247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.022224 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:27.022247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.022236 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-credential-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:27.022247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.022249 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-workload-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:27.024086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.024050 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-kube-api-access-hblkr" (OuterVolumeSpecName: "kube-api-access-hblkr") pod "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" (UID: "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7"). InnerVolumeSpecName "kube-api-access-hblkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:46:27.024086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.024058 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" (UID: "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:46:27.024232 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.024078 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" (UID: "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:46:27.024232 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.024114 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" (UID: "492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:27.123006 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.122971 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblkr\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-kube-api-access-hblkr\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:27.123006 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.122998 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istiod-ca-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:27.123006 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.123008 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-envoy\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:27.123006 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.123016 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:27.123273 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.123024 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7-istio-podinfo\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:27.841179 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.841148 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" Sep 11 18:46:27.841614 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.841195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q" event={"ID":"492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7","Type":"ContainerDied","Data":"de42f52acdd2dc080423b8c55962865cba419c367ca20d8d56a706d5cbd0add4"} Sep 11 18:46:27.841614 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.841253 2579 scope.go:117] "RemoveContainer" containerID="3706159a9fdd3f1b2229daab403351e7028e483d31260769505e805fbb2beb5c" Sep 11 18:46:27.864076 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.864042 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q"] Sep 11 18:46:27.868206 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:27.868180 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72c98d-openshift-default-5d78fd89fb-g8v2q"] Sep 11 18:46:28.229952 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:28.229860 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" path="/var/lib/kubelet/pods/492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7/volumes" Sep 11 18:46:31.648502 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:31.648459 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:46:32.619111 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.619074 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z"] Sep 11 18:46:32.619557 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.619324 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" podUID="73941f7b-2870-4a65-81af-f7dbcb5d6c34" containerName="maas-api" containerID="cri-o://2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2" gracePeriod=30 Sep 11 18:46:32.652407 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.652377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:32.753913 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.753877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:32.856441 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.856403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:32.874467 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.874442 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:46:32.874995 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.874968 2579 generic.go:356] "Generic (PLEG): container finished" podID="73941f7b-2870-4a65-81af-f7dbcb5d6c34" containerID="2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2" exitCode=0 Sep 11 18:46:32.875131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.875045 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" event={"ID":"73941f7b-2870-4a65-81af-f7dbcb5d6c34","Type":"ContainerDied","Data":"2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2"} Sep 11 18:46:32.875131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.875098 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" event={"ID":"73941f7b-2870-4a65-81af-f7dbcb5d6c34","Type":"ContainerDied","Data":"11934b948b1067496b25c1974035deac19ba42b2c3b7ca3d7b522a51d27d379b"} Sep 11 18:46:32.875131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.875127 2579 scope.go:117] "RemoveContainer" containerID="2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2" Sep 11 18:46:32.884616 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.884591 2579 scope.go:117] "RemoveContainer" containerID="2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2" Sep 11 18:46:32.885013 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:46:32.884986 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2\": container with ID starting with 2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2 not found: ID does not exist" containerID="2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2" Sep 11 18:46:32.885100 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.885024 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2"} err="failed to get container status \"2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2\": rpc error: code = NotFound desc = could not find container \"2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2\": container with ID starting with 2228718a294a5d5021250b2d80133448613b1ccb5f36f9792dba3684aee9ecd2 not found: ID does not exist" Sep 11 18:46:32.923879 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.923827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:32.978961 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.978931 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrgvs\" (UniqueName: \"kubernetes.io/projected/73941f7b-2870-4a65-81af-f7dbcb5d6c34-kube-api-access-lrgvs\") pod \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " Sep 11 18:46:32.978961 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.978966 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73941f7b-2870-4a65-81af-f7dbcb5d6c34-openshift-service-ca\") pod \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " Sep 11 18:46:32.979163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.978989 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-metrics-certs\") pod \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " Sep 11 18:46:32.979163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.979008 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-maas-api-tls\") pod \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\" (UID: \"73941f7b-2870-4a65-81af-f7dbcb5d6c34\") " Sep 11 18:46:32.979410 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.979384 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73941f7b-2870-4a65-81af-f7dbcb5d6c34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "73941f7b-2870-4a65-81af-f7dbcb5d6c34" (UID: "73941f7b-2870-4a65-81af-f7dbcb5d6c34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:46:32.981223 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.981196 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "73941f7b-2870-4a65-81af-f7dbcb5d6c34" (UID: "73941f7b-2870-4a65-81af-f7dbcb5d6c34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:46:32.981342 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.981294 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73941f7b-2870-4a65-81af-f7dbcb5d6c34-kube-api-access-lrgvs" (OuterVolumeSpecName: "kube-api-access-lrgvs") pod "73941f7b-2870-4a65-81af-f7dbcb5d6c34" (UID: "73941f7b-2870-4a65-81af-f7dbcb5d6c34"). InnerVolumeSpecName "kube-api-access-lrgvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:46:32.981342 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:32.981297 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "73941f7b-2870-4a65-81af-f7dbcb5d6c34" (UID: "73941f7b-2870-4a65-81af-f7dbcb5d6c34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:46:33.080287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.080240 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrgvs\" (UniqueName: \"kubernetes.io/projected/73941f7b-2870-4a65-81af-f7dbcb5d6c34-kube-api-access-lrgvs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:33.080287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.080284 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73941f7b-2870-4a65-81af-f7dbcb5d6c34-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:33.080287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.080297 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:33.080530 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.080308 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73941f7b-2870-4a65-81af-f7dbcb5d6c34-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:33.203419 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.203329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:33.306463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.306417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:33.881446 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.881417 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z" Sep 11 18:46:33.906321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.906281 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z"] Sep 11 18:46:33.908775 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.908750 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-238bae-6ccd8b8788-8m58z"] Sep 11 18:46:33.916249 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:33.916225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:34.229269 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:34.229185 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73941f7b-2870-4a65-81af-f7dbcb5d6c34" path="/var/lib/kubelet/pods/73941f7b-2870-4a65-81af-f7dbcb5d6c34/volumes" Sep 11 18:46:34.811040 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:34.811007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:37.927846 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:37.927794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:37.990328 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:37.990288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:39.479758 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:39.479719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:39.748020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:39.747926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:39.817716 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:39.817673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:39.981261 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:39.981227 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76"] Sep 11 18:46:39.981456 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:39.981433 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" containerID="cri-o://4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438" gracePeriod=30 Sep 11 18:46:40.008636 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:40.008590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:44.948416 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:44.948376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:45.243598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.243573 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:46:45.384809 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.384766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:45.398850 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.398812 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-envoy\") pod \"c6c95730-6b8e-47f5-8eac-c8c971da7722\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " Sep 11 18:46:45.399045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.398868 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-data\") pod \"c6c95730-6b8e-47f5-8eac-c8c971da7722\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " Sep 11 18:46:45.399045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.398906 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-podinfo\") pod \"c6c95730-6b8e-47f5-8eac-c8c971da7722\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " Sep 11 18:46:45.399045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.398938 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-socket\") pod \"c6c95730-6b8e-47f5-8eac-c8c971da7722\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " Sep 11 18:46:45.399045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.398996 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-certs\") pod \"c6c95730-6b8e-47f5-8eac-c8c971da7722\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " Sep 11 18:46:45.399045 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.399047 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c95730-6b8e-47f5-8eac-c8c971da7722-istiod-ca-cert\") pod \"c6c95730-6b8e-47f5-8eac-c8c971da7722\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " Sep 11 18:46:45.399335 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.399089 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-token\") pod \"c6c95730-6b8e-47f5-8eac-c8c971da7722\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " Sep 11 18:46:45.399335 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.399123 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-credential-socket\") pod \"c6c95730-6b8e-47f5-8eac-c8c971da7722\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " Sep 11 18:46:45.399335 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.399179 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmzkw\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-kube-api-access-zmzkw\") pod \"c6c95730-6b8e-47f5-8eac-c8c971da7722\" (UID: \"c6c95730-6b8e-47f5-8eac-c8c971da7722\") " Sep 11 18:46:45.399522 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.399172 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6c95730-6b8e-47f5-8eac-c8c971da7722" (UID: "c6c95730-6b8e-47f5-8eac-c8c971da7722"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:45.399585 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.399369 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6c95730-6b8e-47f5-8eac-c8c971da7722" (UID: "c6c95730-6b8e-47f5-8eac-c8c971da7722"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:45.399585 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.399435 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6c95730-6b8e-47f5-8eac-c8c971da7722" (UID: "c6c95730-6b8e-47f5-8eac-c8c971da7722"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:45.399585 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.399555 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6c95730-6b8e-47f5-8eac-c8c971da7722" (UID: "c6c95730-6b8e-47f5-8eac-c8c971da7722"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:45.399777 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.399630 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c95730-6b8e-47f5-8eac-c8c971da7722-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6c95730-6b8e-47f5-8eac-c8c971da7722" (UID: "c6c95730-6b8e-47f5-8eac-c8c971da7722"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:46:45.401389 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.401364 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6c95730-6b8e-47f5-8eac-c8c971da7722" (UID: "c6c95730-6b8e-47f5-8eac-c8c971da7722"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:46:45.401510 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.401445 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6c95730-6b8e-47f5-8eac-c8c971da7722" (UID: "c6c95730-6b8e-47f5-8eac-c8c971da7722"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:46:45.401561 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.401530 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6c95730-6b8e-47f5-8eac-c8c971da7722" (UID: "c6c95730-6b8e-47f5-8eac-c8c971da7722"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:46:45.401561 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.401544 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-kube-api-access-zmzkw" (OuterVolumeSpecName: "kube-api-access-zmzkw") pod "c6c95730-6b8e-47f5-8eac-c8c971da7722" (UID: "c6c95730-6b8e-47f5-8eac-c8c971da7722"). InnerVolumeSpecName "kube-api-access-zmzkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:46:45.500195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.500115 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:45.500195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.500144 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c95730-6b8e-47f5-8eac-c8c971da7722-istiod-ca-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:45.500195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.500152 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:45.500195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.500160 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-credential-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:45.500195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.500170 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmzkw\" (UniqueName: \"kubernetes.io/projected/c6c95730-6b8e-47f5-8eac-c8c971da7722-kube-api-access-zmzkw\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:45.500195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.500179 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-envoy\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:45.500195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.500187 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-data\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:45.500195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.500194 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c95730-6b8e-47f5-8eac-c8c971da7722-istio-podinfo\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:45.500195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.500202 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c95730-6b8e-47f5-8eac-c8c971da7722-workload-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:45.639517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639482 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7819-ed97e8"] Sep 11 18:46:45.639934 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639920 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" Sep 11 18:46:45.639988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639936 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" Sep 11 18:46:45.639988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639948 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" containerName="maas-api" Sep 11 18:46:45.639988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639953 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" containerName="maas-api" Sep 11 18:46:45.639988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639958 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" Sep 11 18:46:45.639988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639963 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" Sep 11 18:46:45.639988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639975 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be80f334-e76f-45b8-b3ba-14994c4e9d67" containerName="maas-api" Sep 11 18:46:45.639988 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639980 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be80f334-e76f-45b8-b3ba-14994c4e9d67" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639992 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98b0651-9f57-44b3-871a-89c2ff8c9f4e" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.639998 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98b0651-9f57-44b3-871a-89c2ff8c9f4e" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640006 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640011 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640024 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17297b36-887b-4578-9c36-6c9f3996ba94" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640029 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17297b36-887b-4578-9c36-6c9f3996ba94" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640035 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73941f7b-2870-4a65-81af-f7dbcb5d6c34" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640040 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73941f7b-2870-4a65-81af-f7dbcb5d6c34" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640051 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640056 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640112 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d99ec185-85dd-4ea7-9a4b-eb3cb2c2e84a" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640119 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73941f7b-2870-4a65-81af-f7dbcb5d6c34" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640127 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640132 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17297b36-887b-4578-9c36-6c9f3996ba94" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640138 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640144 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="492f29e9-c3aa-413d-9e32-8e7fd0c1c1e7" containerName="istio-proxy" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640150 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98b0651-9f57-44b3-871a-89c2ff8c9f4e" containerName="maas-api" Sep 11 18:46:45.640228 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.640156 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be80f334-e76f-45b8-b3ba-14994c4e9d67" containerName="maas-api" Sep 11 18:46:45.643475 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.643459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-ed97e8" Sep 11 18:46:45.646192 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.646163 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vvh9z\"" Sep 11 18:46:45.649602 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.649578 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7819-ed97e8"] Sep 11 18:46:45.802529 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.802497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfmc\" (UniqueName: \"kubernetes.io/projected/2d34558a-ba13-4609-9fa1-8ffd42e447bf-kube-api-access-vrfmc\") pod \"test-curl-7819-ed97e8\" (UID: \"2d34558a-ba13-4609-9fa1-8ffd42e447bf\") " pod="opendatahub/test-curl-7819-ed97e8" Sep 11 18:46:45.903216 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.903182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfmc\" (UniqueName: \"kubernetes.io/projected/2d34558a-ba13-4609-9fa1-8ffd42e447bf-kube-api-access-vrfmc\") pod \"test-curl-7819-ed97e8\" (UID: \"2d34558a-ba13-4609-9fa1-8ffd42e447bf\") " pod="opendatahub/test-curl-7819-ed97e8" Sep 11 18:46:45.912381 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.912349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfmc\" (UniqueName: \"kubernetes.io/projected/2d34558a-ba13-4609-9fa1-8ffd42e447bf-kube-api-access-vrfmc\") pod \"test-curl-7819-ed97e8\" (UID: \"2d34558a-ba13-4609-9fa1-8ffd42e447bf\") " pod="opendatahub/test-curl-7819-ed97e8" Sep 11 18:46:45.954725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.954669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-ed97e8" Sep 11 18:46:45.956544 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.956518 2579 generic.go:356] "Generic (PLEG): container finished" podID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerID="4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438" exitCode=0 Sep 11 18:46:45.956664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.956567 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" Sep 11 18:46:45.956664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.956603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" event={"ID":"c6c95730-6b8e-47f5-8eac-c8c971da7722","Type":"ContainerDied","Data":"4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438"} Sep 11 18:46:45.956664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.956656 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76" event={"ID":"c6c95730-6b8e-47f5-8eac-c8c971da7722","Type":"ContainerDied","Data":"ff4dde36e0312a2cb8ed06d1e2be292eea3ce3471fa745d4b67a158ed1512a7f"} Sep 11 18:46:45.956793 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.956681 2579 scope.go:117] "RemoveContainer" containerID="4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438" Sep 11 18:46:45.966427 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.966410 2579 scope.go:117] "RemoveContainer" containerID="0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a" Sep 11 18:46:45.975469 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.975444 2579 scope.go:117] "RemoveContainer" containerID="4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438" Sep 11 18:46:45.975803 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:46:45.975784 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438\": container with ID starting with 4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438 not found: ID does not exist" containerID="4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438" Sep 11 18:46:45.975869 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.975814 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438"} err="failed to get container status \"4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438\": rpc error: code = NotFound desc = could not find container \"4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438\": container with ID starting with 4a6386d23bfefb88265918e000b2cd96906d274512e728ba4d4df5c71820d438 not found: ID does not exist" Sep 11 18:46:45.975869 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.975833 2579 scope.go:117] "RemoveContainer" containerID="0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a" Sep 11 18:46:45.976153 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:46:45.976129 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a\": container with ID starting with 0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a not found: ID does not exist" containerID="0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a" Sep 11 18:46:45.976210 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.976162 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a"} err="failed to get container status \"0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a\": rpc error: code = NotFound desc = could not find container \"0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a\": container with ID starting with 0ddd8a7a3b0ab9c0438e237f48d796c0cde0ceb6fb27f03081220f3c6f88f58a not found: ID does not exist" Sep 11 18:46:45.992110 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.992072 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76"] Sep 11 18:46:45.995931 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:45.995903 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-238bae-openshift-default-7c894fdff8-nqz76"] Sep 11 18:46:46.230550 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:46.230344 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" path="/var/lib/kubelet/pods/c6c95730-6b8e-47f5-8eac-c8c971da7722/volumes" Sep 11 18:46:46.233367 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:46.233346 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7819-ed97e8"] Sep 11 18:46:46.234961 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:46:46.234934 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d34558a_ba13_4609_9fa1_8ffd42e447bf.slice/crio-69f7bfc22b4aabf86d57cc86067a818b537d6d1e4346dcaf17c5be7ac9b1d6b3 WatchSource:0}: Error finding container 69f7bfc22b4aabf86d57cc86067a818b537d6d1e4346dcaf17c5be7ac9b1d6b3: Status 404 returned error can't find the container with id 69f7bfc22b4aabf86d57cc86067a818b537d6d1e4346dcaf17c5be7ac9b1d6b3 Sep 11 18:46:46.963930 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:46.963883 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7819-ed97e8" event={"ID":"2d34558a-ba13-4609-9fa1-8ffd42e447bf","Type":"ContainerStarted","Data":"c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384"} Sep 11 18:46:46.964350 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:46.963938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7819-ed97e8" event={"ID":"2d34558a-ba13-4609-9fa1-8ffd42e447bf","Type":"ContainerStarted","Data":"69f7bfc22b4aabf86d57cc86067a818b537d6d1e4346dcaf17c5be7ac9b1d6b3"} Sep 11 18:46:46.979430 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:46.979376 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7819-ed97e8" podStartSLOduration=1.979360211 podStartE2EDuration="1.979360211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:46:46.977624946 +0000 UTC m=+1875.204301287" watchObservedRunningTime="2026-09-11 18:46:46.979360211 +0000 UTC m=+1875.206036552" Sep 11 18:46:47.577752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:47.577711 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7819-ed97e8"] Sep 11 18:46:47.583980 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:47.583948 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7819-ed97e8"] Sep 11 18:46:47.970831 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:46:47.970797 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7819-ed97e8" container="test-curl-7819-ed97e8" Sep 11 18:46:48.975165 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:48.975128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:48.976604 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:48.976578 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7819-ed97e8" podUID="2d34558a-ba13-4609-9fa1-8ffd42e447bf" containerName="test-curl-7819-ed97e8" containerID="cri-o://c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384" gracePeriod=2 Sep 11 18:46:49.856156 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:49.856120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:46:50.943796 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:50.943761 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:46:51.151438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:51.151410 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-ed97e8" Sep 11 18:46:51.252033 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:51.251999 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrfmc\" (UniqueName: \"kubernetes.io/projected/2d34558a-ba13-4609-9fa1-8ffd42e447bf-kube-api-access-vrfmc\") pod \"2d34558a-ba13-4609-9fa1-8ffd42e447bf\" (UID: \"2d34558a-ba13-4609-9fa1-8ffd42e447bf\") " Sep 11 18:46:51.254089 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:51.254062 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d34558a-ba13-4609-9fa1-8ffd42e447bf-kube-api-access-vrfmc" (OuterVolumeSpecName: "kube-api-access-vrfmc") pod "2d34558a-ba13-4609-9fa1-8ffd42e447bf" (UID: "2d34558a-ba13-4609-9fa1-8ffd42e447bf"). InnerVolumeSpecName "kube-api-access-vrfmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:46:51.353175 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:51.353135 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrfmc\" (UniqueName: \"kubernetes.io/projected/2d34558a-ba13-4609-9fa1-8ffd42e447bf-kube-api-access-vrfmc\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:46:51.996845 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:51.996802 2579 generic.go:356] "Generic (PLEG): container finished" podID="2d34558a-ba13-4609-9fa1-8ffd42e447bf" containerID="c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384" exitCode=137 Sep 11 18:46:51.996845 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:51.996844 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-ed97e8" Sep 11 18:46:51.997350 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:51.996866 2579 scope.go:117] "RemoveContainer" containerID="c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384" Sep 11 18:46:52.006443 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:52.006420 2579 scope.go:117] "RemoveContainer" containerID="c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384" Sep 11 18:46:52.006777 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:46:52.006754 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384\": container with ID starting with c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384 not found: ID does not exist" containerID="c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384" Sep 11 18:46:52.006853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:52.006788 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384"} err="failed to get container status \"c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384\": rpc error: code = NotFound desc = could not find container \"c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384\": container with ID starting with c18de90f157bbb3564d919afb88aec7c88940a49a79da79791f152bcbdf0f384 not found: ID does not exist" Sep 11 18:46:52.007397 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:46:52.007370 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7819-ed97e8\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="2d34558a-ba13-4609-9fa1-8ffd42e447bf" pod="opendatahub/test-curl-7819-ed97e8" Sep 11 18:46:52.233297 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:46:52.233253 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7819-ed97e8\" is forbidden: User \"system:node:ip-10-0-142-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-253.ec2.internal' and this object" podUID="2d34558a-ba13-4609-9fa1-8ffd42e447bf" pod="opendatahub/test-curl-7819-ed97e8" Sep 11 18:46:52.236555 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:46:52.236533 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d34558a-ba13-4609-9fa1-8ffd42e447bf" path="/var/lib/kubelet/pods/2d34558a-ba13-4609-9fa1-8ffd42e447bf/volumes" Sep 11 18:47:00.980229 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:47:00.980132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:47:04.241548 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:47:04.241515 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:47:23.383056 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:47:23.383019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:47:27.369837 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:47:27.369797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:47:41.439849 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:47:41.439808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:47:45.150422 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:47:45.150383 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:48:08.143131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:08.143097 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:48:16.421436 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:16.421377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:16.522860 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:16.522820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:16.709581 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:16.709471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:20.628531 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.628495 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7819-0632fe"] Sep 11 18:48:20.628969 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.628955 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d34558a-ba13-4609-9fa1-8ffd42e447bf" containerName="test-curl-7819-ed97e8" Sep 11 18:48:20.629015 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.628970 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d34558a-ba13-4609-9fa1-8ffd42e447bf" containerName="test-curl-7819-ed97e8" Sep 11 18:48:20.629070 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.629061 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d34558a-ba13-4609-9fa1-8ffd42e447bf" containerName="test-curl-7819-ed97e8" Sep 11 18:48:20.629109 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.629073 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c95730-6b8e-47f5-8eac-c8c971da7722" containerName="istio-proxy" Sep 11 18:48:20.632346 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.632330 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-0632fe" Sep 11 18:48:20.635748 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.635698 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vvh9z\"" Sep 11 18:48:20.638272 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.638248 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7819-0632fe"] Sep 11 18:48:20.788514 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.788473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xz4x\" (UniqueName: \"kubernetes.io/projected/569bc670-8b83-4d36-ad47-582001f99493-kube-api-access-5xz4x\") pod \"test-curl-7819-0632fe\" (UID: \"569bc670-8b83-4d36-ad47-582001f99493\") " pod="opendatahub/test-curl-7819-0632fe" Sep 11 18:48:20.890125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.890020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xz4x\" (UniqueName: \"kubernetes.io/projected/569bc670-8b83-4d36-ad47-582001f99493-kube-api-access-5xz4x\") pod \"test-curl-7819-0632fe\" (UID: \"569bc670-8b83-4d36-ad47-582001f99493\") " pod="opendatahub/test-curl-7819-0632fe" Sep 11 18:48:20.898470 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.898435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xz4x\" (UniqueName: \"kubernetes.io/projected/569bc670-8b83-4d36-ad47-582001f99493-kube-api-access-5xz4x\") pod \"test-curl-7819-0632fe\" (UID: \"569bc670-8b83-4d36-ad47-582001f99493\") " pod="opendatahub/test-curl-7819-0632fe" Sep 11 18:48:20.944225 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:20.944144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-0632fe" Sep 11 18:48:21.219722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:21.219678 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7819-0632fe"] Sep 11 18:48:21.221451 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:48:21.221409 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod569bc670_8b83_4d36_ad47_582001f99493.slice/crio-f0d73e56331edae49353c44afc63beeaf63dd49504774f5b8d1e824b7f392572 WatchSource:0}: Error finding container f0d73e56331edae49353c44afc63beeaf63dd49504774f5b8d1e824b7f392572: Status 404 returned error can't find the container with id f0d73e56331edae49353c44afc63beeaf63dd49504774f5b8d1e824b7f392572 Sep 11 18:48:21.507276 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:21.507241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7819-0632fe" event={"ID":"569bc670-8b83-4d36-ad47-582001f99493","Type":"ContainerStarted","Data":"c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8"} Sep 11 18:48:21.507478 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:21.507289 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7819-0632fe" event={"ID":"569bc670-8b83-4d36-ad47-582001f99493","Type":"ContainerStarted","Data":"f0d73e56331edae49353c44afc63beeaf63dd49504774f5b8d1e824b7f392572"} Sep 11 18:48:21.521463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:21.521415 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7819-0632fe" podStartSLOduration=1.521396599 podStartE2EDuration="1.521396599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:21.520953954 +0000 UTC m=+1969.747630293" watchObservedRunningTime="2026-09-11 18:48:21.521396599 +0000 UTC m=+1969.748072940" Sep 11 18:48:22.104523 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.104477 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7819-0632fe"] Sep 11 18:48:22.107722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.107682 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7819-0632fe"] Sep 11 18:48:22.219176 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.219093 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7819-2023bd"] Sep 11 18:48:22.219543 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.219529 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569bc670-8b83-4d36-ad47-582001f99493" containerName="test-curl-7819-0632fe" Sep 11 18:48:22.219587 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.219546 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="569bc670-8b83-4d36-ad47-582001f99493" containerName="test-curl-7819-0632fe" Sep 11 18:48:22.219631 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.219623 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="569bc670-8b83-4d36-ad47-582001f99493" containerName="test-curl-7819-0632fe" Sep 11 18:48:22.225698 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.225673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-2023bd" Sep 11 18:48:22.234513 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.234408 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7819-2023bd"] Sep 11 18:48:22.305993 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.305956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqf7\" (UniqueName: \"kubernetes.io/projected/9d4f29a3-8797-4959-9056-d3f45a4dc577-kube-api-access-fdqf7\") pod \"test-curl-7819-2023bd\" (UID: \"9d4f29a3-8797-4959-9056-d3f45a4dc577\") " pod="opendatahub/test-curl-7819-2023bd" Sep 11 18:48:22.407437 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.407401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqf7\" (UniqueName: \"kubernetes.io/projected/9d4f29a3-8797-4959-9056-d3f45a4dc577-kube-api-access-fdqf7\") pod \"test-curl-7819-2023bd\" (UID: \"9d4f29a3-8797-4959-9056-d3f45a4dc577\") " pod="opendatahub/test-curl-7819-2023bd" Sep 11 18:48:22.415694 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.415664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqf7\" (UniqueName: \"kubernetes.io/projected/9d4f29a3-8797-4959-9056-d3f45a4dc577-kube-api-access-fdqf7\") pod \"test-curl-7819-2023bd\" (UID: \"9d4f29a3-8797-4959-9056-d3f45a4dc577\") " pod="opendatahub/test-curl-7819-2023bd" Sep 11 18:48:22.513764 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:48:22.513734 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7819-0632fe" container="test-curl-7819-0632fe" Sep 11 18:48:22.537538 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.537511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-2023bd" Sep 11 18:48:22.812447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:22.812421 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7819-2023bd"] Sep 11 18:48:22.813744 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:48:22.813712 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d4f29a3_8797_4959_9056_d3f45a4dc577.slice/crio-28940af0dbf8cf621cbaeb9903654e810bd8b9405291cb33205b8026a6530186 WatchSource:0}: Error finding container 28940af0dbf8cf621cbaeb9903654e810bd8b9405291cb33205b8026a6530186: Status 404 returned error can't find the container with id 28940af0dbf8cf621cbaeb9903654e810bd8b9405291cb33205b8026a6530186 Sep 11 18:48:23.521566 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:23.521527 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7819-2023bd" event={"ID":"9d4f29a3-8797-4959-9056-d3f45a4dc577","Type":"ContainerStarted","Data":"f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b"} Sep 11 18:48:23.522000 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:23.521575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7819-2023bd" event={"ID":"9d4f29a3-8797-4959-9056-d3f45a4dc577","Type":"ContainerStarted","Data":"28940af0dbf8cf621cbaeb9903654e810bd8b9405291cb33205b8026a6530186"} Sep 11 18:48:23.522000 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:23.521630 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7819-0632fe" podUID="569bc670-8b83-4d36-ad47-582001f99493" containerName="test-curl-7819-0632fe" containerID="cri-o://c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8" gracePeriod=2 Sep 11 18:48:23.536698 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:23.536652 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7819-2023bd" podStartSLOduration=1.536638824 podStartE2EDuration="1.536638824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:23.535340765 +0000 UTC m=+1971.762017106" watchObservedRunningTime="2026-09-11 18:48:23.536638824 +0000 UTC m=+1971.763315164" Sep 11 18:48:24.114610 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:24.114569 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7819-2023bd"] Sep 11 18:48:24.118896 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:24.118866 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7819-2023bd"] Sep 11 18:48:24.145087 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:24.145055 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:48:24.527642 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:48:24.527568 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7819-2023bd" container="test-curl-7819-2023bd" Sep 11 18:48:25.532675 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:25.532638 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7819-2023bd" podUID="9d4f29a3-8797-4959-9056-d3f45a4dc577" containerName="test-curl-7819-2023bd" containerID="cri-o://f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b" gracePeriod=2 Sep 11 18:48:25.700963 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:25.700935 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-0632fe" Sep 11 18:48:25.842679 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:25.842596 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xz4x\" (UniqueName: \"kubernetes.io/projected/569bc670-8b83-4d36-ad47-582001f99493-kube-api-access-5xz4x\") pod \"569bc670-8b83-4d36-ad47-582001f99493\" (UID: \"569bc670-8b83-4d36-ad47-582001f99493\") " Sep 11 18:48:25.844821 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:25.844786 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569bc670-8b83-4d36-ad47-582001f99493-kube-api-access-5xz4x" (OuterVolumeSpecName: "kube-api-access-5xz4x") pod "569bc670-8b83-4d36-ad47-582001f99493" (UID: "569bc670-8b83-4d36-ad47-582001f99493"). InnerVolumeSpecName "kube-api-access-5xz4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:48:25.944164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:25.944132 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xz4x\" (UniqueName: \"kubernetes.io/projected/569bc670-8b83-4d36-ad47-582001f99493-kube-api-access-5xz4x\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:48:26.088847 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:26.088811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:26.229672 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:26.229593 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569bc670-8b83-4d36-ad47-582001f99493" path="/var/lib/kubelet/pods/569bc670-8b83-4d36-ad47-582001f99493/volumes" Sep 11 18:48:26.539524 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:26.539484 2579 generic.go:356] "Generic (PLEG): container finished" podID="569bc670-8b83-4d36-ad47-582001f99493" containerID="c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8" exitCode=137 Sep 11 18:48:26.540021 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:26.539529 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-0632fe" Sep 11 18:48:26.540021 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:26.539547 2579 scope.go:117] "RemoveContainer" containerID="c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8" Sep 11 18:48:26.548447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:26.548424 2579 scope.go:117] "RemoveContainer" containerID="c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8" Sep 11 18:48:26.548760 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:48:26.548738 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8\": container with ID starting with c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8 not found: ID does not exist" containerID="c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8" Sep 11 18:48:26.548822 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:26.548771 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8"} err="failed to get container status \"c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8\": rpc error: code = NotFound desc = could not find container \"c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8\": container with ID starting with c469b502133ca740bd31c1a0a58bb7930e6b63cba4339b9aa47e26875cc087c8 not found: ID does not exist" Sep 11 18:48:27.065681 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:27.065644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:27.711966 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:27.711937 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-2023bd" Sep 11 18:48:27.864369 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:27.864281 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdqf7\" (UniqueName: \"kubernetes.io/projected/9d4f29a3-8797-4959-9056-d3f45a4dc577-kube-api-access-fdqf7\") pod \"9d4f29a3-8797-4959-9056-d3f45a4dc577\" (UID: \"9d4f29a3-8797-4959-9056-d3f45a4dc577\") " Sep 11 18:48:27.866491 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:27.866452 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4f29a3-8797-4959-9056-d3f45a4dc577-kube-api-access-fdqf7" (OuterVolumeSpecName: "kube-api-access-fdqf7") pod "9d4f29a3-8797-4959-9056-d3f45a4dc577" (UID: "9d4f29a3-8797-4959-9056-d3f45a4dc577"). InnerVolumeSpecName "kube-api-access-fdqf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:48:27.965977 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:27.965943 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdqf7\" (UniqueName: \"kubernetes.io/projected/9d4f29a3-8797-4959-9056-d3f45a4dc577-kube-api-access-fdqf7\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:48:28.229410 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:28.229329 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4f29a3-8797-4959-9056-d3f45a4dc577" path="/var/lib/kubelet/pods/9d4f29a3-8797-4959-9056-d3f45a4dc577/volumes" Sep 11 18:48:28.555487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:28.555448 2579 generic.go:356] "Generic (PLEG): container finished" podID="9d4f29a3-8797-4959-9056-d3f45a4dc577" containerID="f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b" exitCode=137 Sep 11 18:48:28.555487 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:28.555489 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7819-2023bd" Sep 11 18:48:28.555696 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:28.555537 2579 scope.go:117] "RemoveContainer" containerID="f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b" Sep 11 18:48:28.564447 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:28.564432 2579 scope.go:117] "RemoveContainer" containerID="f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b" Sep 11 18:48:28.564741 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:48:28.564721 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b\": container with ID starting with f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b not found: ID does not exist" containerID="f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b" Sep 11 18:48:28.564786 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:28.564749 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b"} err="failed to get container status \"f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b\": rpc error: code = NotFound desc = could not find container \"f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b\": container with ID starting with f263431a034138648852e11f6e6cd545816fbece423363706aecdb589144751b not found: ID does not exist" Sep 11 18:48:34.546721 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:34.546679 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:48:35.346718 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:35.346667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:43.248317 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:43.248283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:43.349556 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:43.349525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:43.437752 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:48:43.437716 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:49:20.169532 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:49:20.169496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:23.942077 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:49:23.942042 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:49:44.989313 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:49:44.989277 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:50:00.349106 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:00.349014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:50:05.605853 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:05.605805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:06.547763 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:06.547731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:06.983628 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:06.983519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:07.317153 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:07.317097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:07.421564 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:07.421526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:09.653161 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:09.653116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:10.361912 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:10.361867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:10.461764 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:10.461694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:10.587956 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:10.587906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:11.459840 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:11.459803 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:50:11.638785 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:11.638748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:12.695046 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.695016 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz"] Sep 11 18:50:12.695657 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.695622 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4f29a3-8797-4959-9056-d3f45a4dc577" containerName="test-curl-7819-2023bd" Sep 11 18:50:12.695657 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.695656 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4f29a3-8797-4959-9056-d3f45a4dc577" containerName="test-curl-7819-2023bd" Sep 11 18:50:12.695870 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.695792 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4f29a3-8797-4959-9056-d3f45a4dc577" containerName="test-curl-7819-2023bd" Sep 11 18:50:12.699077 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.699049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.701565 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.701545 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-da4d13-serving-cert\"" Sep 11 18:50:12.709791 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.709766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dfdea1-ca52-417a-b051-d31dd980c720-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.709910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.709823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-maas-api-tls\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.709910 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.709871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-metrics-certs\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.710026 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.709974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdf46\" (UniqueName: \"kubernetes.io/projected/a3dfdea1-ca52-417a-b051-d31dd980c720-kube-api-access-pdf46\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.711666 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.711642 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz"] Sep 11 18:50:12.738065 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.738032 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:12.811412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.811379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-maas-api-tls\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.811412 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.811414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-metrics-certs\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.811674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.811469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdf46\" (UniqueName: \"kubernetes.io/projected/a3dfdea1-ca52-417a-b051-d31dd980c720-kube-api-access-pdf46\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.811674 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.811503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dfdea1-ca52-417a-b051-d31dd980c720-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.811674 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:50:12.811519 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-serving-cert: secret "maas-api-e2e-ipp-a-da4d13-serving-cert" not found Sep 11 18:50:12.811674 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:50:12.811591 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-maas-api-tls podName:a3dfdea1-ca52-417a-b051-d31dd980c720 nodeName:}" failed. No retries permitted until 2026-09-11 18:50:13.311573999 +0000 UTC m=+2081.538250322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-maas-api-tls") pod "maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" (UID: "a3dfdea1-ca52-417a-b051-d31dd980c720") : secret "maas-api-e2e-ipp-a-da4d13-serving-cert" not found Sep 11 18:50:12.812109 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.812091 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dfdea1-ca52-417a-b051-d31dd980c720-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.813968 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.813941 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-metrics-certs\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.820926 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.820908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdf46\" (UniqueName: \"kubernetes.io/projected/a3dfdea1-ca52-417a-b051-d31dd980c720-kube-api-access-pdf46\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:12.839500 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:12.839468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:13.159984 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:13.159949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:13.260413 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:13.260375 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:13.315929 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:13.315897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-maas-api-tls\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:13.318459 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:13.318435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-maas-api-tls\") pod \"maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:13.353140 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:13.353107 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:13.610904 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:13.610863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:13.887982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:13.887947 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz"] Sep 11 18:50:13.890016 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:50:13.889986 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dfdea1_ca52_417a_b051_d31dd980c720.slice/crio-41ce3f22f32399c3b70e0062f6c261d0733fae86ecbb5e02ed7131bca5bbdfdc WatchSource:0}: Error finding container 41ce3f22f32399c3b70e0062f6c261d0733fae86ecbb5e02ed7131bca5bbdfdc: Status 404 returned error can't find the container with id 41ce3f22f32399c3b70e0062f6c261d0733fae86ecbb5e02ed7131bca5bbdfdc Sep 11 18:50:13.891431 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:13.891414 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:50:14.161134 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:14.161037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" event={"ID":"a3dfdea1-ca52-417a-b051-d31dd980c720","Type":"ContainerStarted","Data":"41ce3f22f32399c3b70e0062f6c261d0733fae86ecbb5e02ed7131bca5bbdfdc"} Sep 11 18:50:15.171479 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:15.171421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" event={"ID":"a3dfdea1-ca52-417a-b051-d31dd980c720","Type":"ContainerStarted","Data":"0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba"} Sep 11 18:50:15.171914 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:15.171653 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:15.188665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:15.188612 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" podStartSLOduration=2.698226569 podStartE2EDuration="3.188597135s" totalImagesPullingTime="490.370566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:15.188108572 +0000 UTC m=+2083.414784929" watchObservedRunningTime="2026-09-11 18:50:15.188597135 +0000 UTC m=+2083.415273475" Sep 11 18:50:21.182327 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:21.182295 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:50:21.742453 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:21.742422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:50:32.433951 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:32.433920 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:50:32.449370 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:32.449342 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:50:33.575545 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:33.575504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:33.676395 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:33.676355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:34.739473 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:34.739438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:35.317824 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:35.317784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:35.500022 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:35.499969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:35.994302 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:35.994268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:36.302404 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:36.302362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:36.400287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:36.400251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:36.529734 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:36.529678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:37.287330 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:37.287280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:38.819319 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:38.819280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:39.561445 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:39.561404 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:39.663462 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:39.663431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:39.789541 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:39.789505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:40.719601 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:40.719561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:41.634309 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:41.634275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:42.424954 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.424920 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq"] Sep 11 18:50:42.428681 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.428660 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.430995 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.430970 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0ed2c8-serving-cert\"" Sep 11 18:50:42.442230 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.442166 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq"] Sep 11 18:50:42.459994 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.459966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:42.566313 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.566277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:42.596462 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.596431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmg4f\" (UniqueName: \"kubernetes.io/projected/a510cd4b-c710-4740-bc15-655db68cc15f-kube-api-access-fmg4f\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.596618 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.596469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.596618 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.596499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.596693 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.596611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510cd4b-c710-4740-bc15-655db68cc15f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.697611 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.697507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.697611 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.697548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.697611 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.697601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510cd4b-c710-4740-bc15-655db68cc15f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.697949 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.697734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmg4f\" (UniqueName: \"kubernetes.io/projected/a510cd4b-c710-4740-bc15-655db68cc15f-kube-api-access-fmg4f\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.697949 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:50:42.697778 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-serving-cert: secret "maas-api-e2e-ipp-b-0ed2c8-serving-cert" not found Sep 11 18:50:42.697949 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:50:42.697874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-maas-api-tls podName:a510cd4b-c710-4740-bc15-655db68cc15f nodeName:}" failed. No retries permitted until 2026-09-11 18:50:43.197853748 +0000 UTC m=+2111.424530081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-maas-api-tls") pod "maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" (UID: "a510cd4b-c710-4740-bc15-655db68cc15f") : secret "maas-api-e2e-ipp-b-0ed2c8-serving-cert" not found Sep 11 18:50:42.698377 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.698356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510cd4b-c710-4740-bc15-655db68cc15f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.700192 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.700169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.706164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.706146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmg4f\" (UniqueName: \"kubernetes.io/projected/a510cd4b-c710-4740-bc15-655db68cc15f-kube-api-access-fmg4f\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:42.953012 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:42.952909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:43.058299 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:43.058255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:43.156027 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:43.155988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:43.204000 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:43.203909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:43.206490 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:43.206470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:43.343002 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:43.342970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:43.619276 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:43.619250 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq"] Sep 11 18:50:43.621146 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:50:43.621118 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda510cd4b_c710_4740_bc15_655db68cc15f.slice/crio-2cf4a700febc411313fc1a4be849bee8785f18bc13c1da42856f855ca8c5c5f6 WatchSource:0}: Error finding container 2cf4a700febc411313fc1a4be849bee8785f18bc13c1da42856f855ca8c5c5f6: Status 404 returned error can't find the container with id 2cf4a700febc411313fc1a4be849bee8785f18bc13c1da42856f855ca8c5c5f6 Sep 11 18:50:44.336118 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:44.336079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" event={"ID":"a510cd4b-c710-4740-bc15-655db68cc15f","Type":"ContainerStarted","Data":"a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441"} Sep 11 18:50:44.336310 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:44.336146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" event={"ID":"a510cd4b-c710-4740-bc15-655db68cc15f","Type":"ContainerStarted","Data":"2cf4a700febc411313fc1a4be849bee8785f18bc13c1da42856f855ca8c5c5f6"} Sep 11 18:50:44.336310 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:44.336176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:50:44.357598 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:44.357545 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" podStartSLOduration=1.980785089 podStartE2EDuration="2.357527876s" totalImagesPullingTime="376.742787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:44.353758752 +0000 UTC m=+2112.580435109" watchObservedRunningTime="2026-09-11 18:50:44.357527876 +0000 UTC m=+2112.584204276" Sep 11 18:50:46.855913 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:46.855873 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:50:50.347834 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:50:50.347805 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:51:04.490131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:04.490093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:04.590698 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:04.590656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:05.234695 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:05.234659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:05.995189 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:05.995140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:06.698637 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:06.698598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:07.374074 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:07.374041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:16.286890 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:16.286849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:25.672741 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:25.672634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:29.744645 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:29.744610 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:51:44.142759 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:44.142721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:51:46.038020 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:46.037982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:51.021462 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.021400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:51.310505 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.310422 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96"] Sep 11 18:51:51.314233 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.314211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.317794 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.317758 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1cb71b-openshift-default-dockercfg-9ssfn\"" Sep 11 18:51:51.317938 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.317758 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 18:51:51.327999 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.327972 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96"] Sep 11 18:51:51.347057 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.346996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:51.449755 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.449693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9262010-ced6-42e0-842c-3a266af78379-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.449948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.449772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-certs\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.449948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.449804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-istio-token\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.449948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.449830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9262010-ced6-42e0-842c-3a266af78379-istio-podinfo\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.449948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.449888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-credential-socket\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.449948 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.449937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-envoy\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.450154 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.449989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvgm\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-kube-api-access-rgvgm\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.450154 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.450009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-socket\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.450154 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.450051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-data\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551198 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvgm\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-kube-api-access-rgvgm\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551198 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-socket\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-data\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9262010-ced6-42e0-842c-3a266af78379-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551463 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-certs\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551623 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-istio-token\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551623 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9262010-ced6-42e0-842c-3a266af78379-istio-podinfo\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551623 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-credential-socket\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551623 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-envoy\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551918 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551732 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-socket\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551918 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-data\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551918 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551844 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-certs\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.551918 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.551905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-credential-socket\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.552157 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.552134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9262010-ced6-42e0-842c-3a266af78379-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.553819 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.553798 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-envoy\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.554107 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.554089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9262010-ced6-42e0-842c-3a266af78379-istio-podinfo\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.559086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.559065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-istio-token\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.559262 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.559244 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvgm\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-kube-api-access-rgvgm\") pod \"e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.626276 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.626186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:51.664530 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.664490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:51.904755 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.904726 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96"] Sep 11 18:51:51.906859 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:51:51.906826 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9262010_ced6_42e0_842c_3a266af78379.slice/crio-6761367c7fa1fa59dfb7eace9a2025ff6b94564fc719d3dd9fb22eadbc1383fb WatchSource:0}: Error finding container 6761367c7fa1fa59dfb7eace9a2025ff6b94564fc719d3dd9fb22eadbc1383fb: Status 404 returned error can't find the container with id 6761367c7fa1fa59dfb7eace9a2025ff6b94564fc719d3dd9fb22eadbc1383fb Sep 11 18:51:51.909837 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.909806 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:51:51.909956 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.909886 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:51:51.909956 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:51.909935 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:51:52.736563 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:52.736520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" event={"ID":"c9262010-ced6-42e0-842c-3a266af78379","Type":"ContainerStarted","Data":"9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b"} Sep 11 18:51:52.736983 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:52.736585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" event={"ID":"c9262010-ced6-42e0-842c-3a266af78379","Type":"ContainerStarted","Data":"6761367c7fa1fa59dfb7eace9a2025ff6b94564fc719d3dd9fb22eadbc1383fb"} Sep 11 18:51:52.759664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:52.759607 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" podStartSLOduration=1.759589233 podStartE2EDuration="1.759589233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:51:52.757732591 +0000 UTC m=+2180.984408942" watchObservedRunningTime="2026-09-11 18:51:52.759589233 +0000 UTC m=+2180.986265573" Sep 11 18:51:53.627075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:53.627040 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:53.632352 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:53.632326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:53.743542 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:53.743502 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:53.744428 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:53.744410 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:51:53.940231 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:53.940145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:54.718678 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:54.718644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:54.873123 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:54.873048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:55.535619 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:55.535560 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:56.321157 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:56.321120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:57.221677 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:57.221633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:57.913908 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:57.913869 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls"] Sep 11 18:51:57.917633 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:57.917617 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:57.920800 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:57.920776 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-1cb71b-serving-cert\"" Sep 11 18:51:57.936645 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:57.936621 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls"] Sep 11 18:51:58.013805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.013757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:58.014004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.013857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.014004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.013911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.014004 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.013941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fe228-04ee-46ec-87b4-6b497d88fab1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.014121 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.014054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4cb\" (UniqueName: \"kubernetes.io/projected/864fe228-04ee-46ec-87b4-6b497d88fab1-kube-api-access-st4cb\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.114868 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.114825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4cb\" (UniqueName: \"kubernetes.io/projected/864fe228-04ee-46ec-87b4-6b497d88fab1-kube-api-access-st4cb\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.114868 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.114870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.115142 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.114899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.115142 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.114944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fe228-04ee-46ec-87b4-6b497d88fab1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.115142 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:51:58.115003 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-serving-cert: secret "maas-api-e2e-ipp-cleanup-1cb71b-serving-cert" not found Sep 11 18:51:58.115142 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:51:58.115077 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-maas-api-tls podName:864fe228-04ee-46ec-87b4-6b497d88fab1 nodeName:}" failed. No retries permitted until 2026-09-11 18:51:58.615057438 +0000 UTC m=+2186.841733764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" (UID: "864fe228-04ee-46ec-87b4-6b497d88fab1") : secret "maas-api-e2e-ipp-cleanup-1cb71b-serving-cert" not found Sep 11 18:51:58.115639 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.115613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fe228-04ee-46ec-87b4-6b497d88fab1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.117593 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.117567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.118000 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.117974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:58.128364 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.128343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4cb\" (UniqueName: \"kubernetes.io/projected/864fe228-04ee-46ec-87b4-6b497d88fab1-kube-api-access-st4cb\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.444063 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.444025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:58.554609 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.554567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:58.620535 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.620495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.622893 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.622868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:58.659005 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.658972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:58.829017 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:58.828984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:59.103629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:59.103551 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls"] Sep 11 18:51:59.106830 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:51:59.106804 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864fe228_04ee_46ec_87b4_6b497d88fab1.slice/crio-68279e64b8929bf9c62a542761bd6acac7347a29b4f3d3452b15ad95749225ed WatchSource:0}: Error finding container 68279e64b8929bf9c62a542761bd6acac7347a29b4f3d3452b15ad95749225ed: Status 404 returned error can't find the container with id 68279e64b8929bf9c62a542761bd6acac7347a29b4f3d3452b15ad95749225ed Sep 11 18:51:59.784150 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:59.784112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" event={"ID":"864fe228-04ee-46ec-87b4-6b497d88fab1","Type":"ContainerStarted","Data":"32e820b48fe4ecfde7a99f7fb90cd6adb47a21ea54ac612420e57110db216156"} Sep 11 18:51:59.784360 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:59.784210 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" event={"ID":"864fe228-04ee-46ec-87b4-6b497d88fab1","Type":"ContainerStarted","Data":"68279e64b8929bf9c62a542761bd6acac7347a29b4f3d3452b15ad95749225ed"} Sep 11 18:51:59.784360 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:59.784330 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:51:59.803742 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:51:59.803674 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" podStartSLOduration=2.448857415 podStartE2EDuration="2.803658116s" totalImagesPullingTime="354.800701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:51:59.8013331 +0000 UTC m=+2188.028009468" watchObservedRunningTime="2026-09-11 18:51:59.803658116 +0000 UTC m=+2188.030334456" Sep 11 18:52:04.414009 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:04.413971 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:52:05.795938 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:05.795908 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:52:14.884154 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:14.884044 2579 generic.go:356] "Generic (PLEG): container finished" podID="c9262010-ced6-42e0-842c-3a266af78379" containerID="9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b" exitCode=0 Sep 11 18:52:14.884154 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:14.884121 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" event={"ID":"c9262010-ced6-42e0-842c-3a266af78379","Type":"ContainerDied","Data":"9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b"} Sep 11 18:52:14.884655 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:14.884623 2579 scope.go:117] "RemoveContainer" containerID="9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b" Sep 11 18:52:14.885847 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:14.885819 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:52:14.885908 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:14.885888 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:52:14.885944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:14.885916 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 18:52:15.897176 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:15.897132 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" event={"ID":"c9262010-ced6-42e0-842c-3a266af78379","Type":"ContainerStarted","Data":"a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5"} Sep 11 18:52:16.626539 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:16.626504 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:52:16.626802 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:16.626776 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 18:52:16.626911 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:16.626835 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 18:52:17.626939 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:17.626901 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 18:52:17.627339 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:17.626961 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 18:52:18.627472 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:18.627432 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 18:52:18.627884 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:18.627486 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 18:52:19.626676 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:19.626638 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 18:52:19.626884 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:19.626695 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 18:52:20.013477 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:20.013446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:20.627508 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:20.627468 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 18:52:20.627728 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:20.627525 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 18:52:20.830247 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:20.830203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:21.506304 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:21.506261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:21.630937 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:21.630907 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:52:21.631186 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:21.631169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:52:21.631780 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:21.631762 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:52:22.312544 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:22.312506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:22.986652 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:22.986610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:23.626813 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:23.626773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:24.757360 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:24.757320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:26.208918 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:26.208882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:26.870312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:26.870276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:27.642149 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:27.642095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:28.388307 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:28.388271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:29.253502 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:29.253471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:30.067906 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:30.067872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:36.813920 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:36.813886 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls"] Sep 11 18:52:36.814410 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:36.814072 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" podUID="864fe228-04ee-46ec-87b4-6b497d88fab1" containerName="maas-api" containerID="cri-o://32e820b48fe4ecfde7a99f7fb90cd6adb47a21ea54ac612420e57110db216156" gracePeriod=30 Sep 11 18:52:36.889905 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:36.889867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:37.031403 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.031367 2579 generic.go:356] "Generic (PLEG): container finished" podID="864fe228-04ee-46ec-87b4-6b497d88fab1" containerID="32e820b48fe4ecfde7a99f7fb90cd6adb47a21ea54ac612420e57110db216156" exitCode=0 Sep 11 18:52:37.031587 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.031440 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" event={"ID":"864fe228-04ee-46ec-87b4-6b497d88fab1","Type":"ContainerDied","Data":"32e820b48fe4ecfde7a99f7fb90cd6adb47a21ea54ac612420e57110db216156"} Sep 11 18:52:37.082932 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.082905 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:52:37.090531 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.090499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:37.213725 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.213673 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-metrics-certs\") pod \"864fe228-04ee-46ec-87b4-6b497d88fab1\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " Sep 11 18:52:37.213871 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.213739 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st4cb\" (UniqueName: \"kubernetes.io/projected/864fe228-04ee-46ec-87b4-6b497d88fab1-kube-api-access-st4cb\") pod \"864fe228-04ee-46ec-87b4-6b497d88fab1\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " Sep 11 18:52:37.213871 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.213775 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fe228-04ee-46ec-87b4-6b497d88fab1-openshift-service-ca\") pod \"864fe228-04ee-46ec-87b4-6b497d88fab1\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " Sep 11 18:52:37.213959 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.213918 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-maas-api-tls\") pod \"864fe228-04ee-46ec-87b4-6b497d88fab1\" (UID: \"864fe228-04ee-46ec-87b4-6b497d88fab1\") " Sep 11 18:52:37.214223 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.214200 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864fe228-04ee-46ec-87b4-6b497d88fab1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "864fe228-04ee-46ec-87b4-6b497d88fab1" (UID: "864fe228-04ee-46ec-87b4-6b497d88fab1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:52:37.215914 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.215885 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864fe228-04ee-46ec-87b4-6b497d88fab1-kube-api-access-st4cb" (OuterVolumeSpecName: "kube-api-access-st4cb") pod "864fe228-04ee-46ec-87b4-6b497d88fab1" (UID: "864fe228-04ee-46ec-87b4-6b497d88fab1"). InnerVolumeSpecName "kube-api-access-st4cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:52:37.215914 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.215895 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "864fe228-04ee-46ec-87b4-6b497d88fab1" (UID: "864fe228-04ee-46ec-87b4-6b497d88fab1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:52:37.216051 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.215912 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "864fe228-04ee-46ec-87b4-6b497d88fab1" (UID: "864fe228-04ee-46ec-87b4-6b497d88fab1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:52:37.314827 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.314793 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:37.314827 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.314826 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st4cb\" (UniqueName: \"kubernetes.io/projected/864fe228-04ee-46ec-87b4-6b497d88fab1-kube-api-access-st4cb\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:37.314827 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.314836 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fe228-04ee-46ec-87b4-6b497d88fab1-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:37.315056 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.314845 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fe228-04ee-46ec-87b4-6b497d88fab1-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:37.337749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.337647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:37.486162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.486113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:37.614996 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.614910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:37.995112 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:37.995018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:38.038982 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:38.038947 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" Sep 11 18:52:38.039160 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:38.038984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls" event={"ID":"864fe228-04ee-46ec-87b4-6b497d88fab1","Type":"ContainerDied","Data":"68279e64b8929bf9c62a542761bd6acac7347a29b4f3d3452b15ad95749225ed"} Sep 11 18:52:38.039160 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:38.039053 2579 scope.go:117] "RemoveContainer" containerID="32e820b48fe4ecfde7a99f7fb90cd6adb47a21ea54ac612420e57110db216156" Sep 11 18:52:38.072038 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:38.072007 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls"] Sep 11 18:52:38.082312 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:38.082284 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1cb71b-548d9bf9f-dhmls"] Sep 11 18:52:38.116852 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:38.116816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:38.230377 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:38.230346 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864fe228-04ee-46ec-87b4-6b497d88fab1" path="/var/lib/kubelet/pods/864fe228-04ee-46ec-87b4-6b497d88fab1/volumes" Sep 11 18:52:39.250828 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:39.250790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:40.424794 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:40.424747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:41.186329 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:41.186297 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:52:41.652132 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:41.652089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:42.458401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:42.458364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:44.354241 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:44.354201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:44.455676 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:44.455640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:44.613358 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:44.613266 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96"] Sep 11 18:52:44.613567 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:44.613534 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" containerID="cri-o://a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5" gracePeriod=30 Sep 11 18:52:44.645953 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:44.645917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:48.550262 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:48.550200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:49.430891 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.430857 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:49.869808 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.869778 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:52:49.927080 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927049 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-envoy\") pod \"c9262010-ced6-42e0-842c-3a266af78379\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " Sep 11 18:52:49.927271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927086 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-certs\") pod \"c9262010-ced6-42e0-842c-3a266af78379\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " Sep 11 18:52:49.927271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927110 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-data\") pod \"c9262010-ced6-42e0-842c-3a266af78379\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " Sep 11 18:52:49.927271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-credential-socket\") pod \"c9262010-ced6-42e0-842c-3a266af78379\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " Sep 11 18:52:49.927271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927198 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9262010-ced6-42e0-842c-3a266af78379-istio-podinfo\") pod \"c9262010-ced6-42e0-842c-3a266af78379\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " Sep 11 18:52:49.927271 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9262010-ced6-42e0-842c-3a266af78379-istiod-ca-cert\") pod \"c9262010-ced6-42e0-842c-3a266af78379\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " Sep 11 18:52:49.927551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927308 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-istio-token\") pod \"c9262010-ced6-42e0-842c-3a266af78379\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " Sep 11 18:52:49.927551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927352 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-socket\") pod \"c9262010-ced6-42e0-842c-3a266af78379\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " Sep 11 18:52:49.927551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927361 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c9262010-ced6-42e0-842c-3a266af78379" (UID: "c9262010-ced6-42e0-842c-3a266af78379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:52:49.927551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927440 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-data" (OuterVolumeSpecName: "istio-data") pod "c9262010-ced6-42e0-842c-3a266af78379" (UID: "c9262010-ced6-42e0-842c-3a266af78379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:52:49.927551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927457 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgvgm\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-kube-api-access-rgvgm\") pod \"c9262010-ced6-42e0-842c-3a266af78379\" (UID: \"c9262010-ced6-42e0-842c-3a266af78379\") " Sep 11 18:52:49.927551 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927464 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c9262010-ced6-42e0-842c-3a266af78379" (UID: "c9262010-ced6-42e0-842c-3a266af78379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:52:49.927900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927634 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9262010-ced6-42e0-842c-3a266af78379-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c9262010-ced6-42e0-842c-3a266af78379" (UID: "c9262010-ced6-42e0-842c-3a266af78379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:52:49.927900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927813 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:49.927900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927832 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-data\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:49.927900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927844 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-credential-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:49.927900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927857 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9262010-ced6-42e0-842c-3a266af78379-istiod-ca-cert\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:49.927900 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.927861 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c9262010-ced6-42e0-842c-3a266af78379" (UID: "c9262010-ced6-42e0-842c-3a266af78379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:52:49.929491 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.929460 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9262010-ced6-42e0-842c-3a266af78379-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c9262010-ced6-42e0-842c-3a266af78379" (UID: "c9262010-ced6-42e0-842c-3a266af78379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:52:49.929616 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.929568 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-kube-api-access-rgvgm" (OuterVolumeSpecName: "kube-api-access-rgvgm") pod "c9262010-ced6-42e0-842c-3a266af78379" (UID: "c9262010-ced6-42e0-842c-3a266af78379"). InnerVolumeSpecName "kube-api-access-rgvgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:52:49.929689 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.929669 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c9262010-ced6-42e0-842c-3a266af78379" (UID: "c9262010-ced6-42e0-842c-3a266af78379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:52:49.929800 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:49.929744 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-istio-token" (OuterVolumeSpecName: "istio-token") pod "c9262010-ced6-42e0-842c-3a266af78379" (UID: "c9262010-ced6-42e0-842c-3a266af78379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:52:50.028327 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.028291 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-workload-socket\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:50.028327 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.028325 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgvgm\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-kube-api-access-rgvgm\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:50.028536 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.028335 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9262010-ced6-42e0-842c-3a266af78379-istio-envoy\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:50.028536 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.028344 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9262010-ced6-42e0-842c-3a266af78379-istio-podinfo\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:50.028536 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.028352 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9262010-ced6-42e0-842c-3a266af78379-istio-token\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:52:50.118024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.117990 2579 generic.go:356] "Generic (PLEG): container finished" podID="c9262010-ced6-42e0-842c-3a266af78379" containerID="a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5" exitCode=0 Sep 11 18:52:50.118195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.118036 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" Sep 11 18:52:50.118195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.118078 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" event={"ID":"c9262010-ced6-42e0-842c-3a266af78379","Type":"ContainerDied","Data":"a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5"} Sep 11 18:52:50.118195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.118156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96" event={"ID":"c9262010-ced6-42e0-842c-3a266af78379","Type":"ContainerDied","Data":"6761367c7fa1fa59dfb7eace9a2025ff6b94564fc719d3dd9fb22eadbc1383fb"} Sep 11 18:52:50.118195 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.118178 2579 scope.go:117] "RemoveContainer" containerID="a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5" Sep 11 18:52:50.128370 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.128349 2579 scope.go:117] "RemoveContainer" containerID="9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b" Sep 11 18:52:50.136995 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.136977 2579 scope.go:117] "RemoveContainer" containerID="a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5" Sep 11 18:52:50.137287 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:52:50.137266 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5\": container with ID starting with a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5 not found: ID does not exist" containerID="a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5" Sep 11 18:52:50.137382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.137298 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5"} err="failed to get container status \"a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5\": rpc error: code = NotFound desc = could not find container \"a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5\": container with ID starting with a701222c43a14ab5758e08188a0ee1afe2ce8fad971b444b8b033734ef3728e5 not found: ID does not exist" Sep 11 18:52:50.137382 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.137322 2579 scope.go:117] "RemoveContainer" containerID="9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b" Sep 11 18:52:50.137613 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:52:50.137594 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b\": container with ID starting with 9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b not found: ID does not exist" containerID="9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b" Sep 11 18:52:50.137650 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.137619 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b"} err="failed to get container status \"9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b\": rpc error: code = NotFound desc = could not find container \"9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b\": container with ID starting with 9f818f1e44d55e399050391a965d22763725545bd034cd176cba9c45189bee9b not found: ID does not exist" Sep 11 18:52:50.157843 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.157813 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96"] Sep 11 18:52:50.172131 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.172104 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1cb71b-openshift-default-5d577868dd-j9z96"] Sep 11 18:52:50.230096 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.230056 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9262010-ced6-42e0-842c-3a266af78379" path="/var/lib/kubelet/pods/c9262010-ced6-42e0-842c-3a266af78379/volumes" Sep 11 18:52:50.305605 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:50.305484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:51.161946 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:51.161912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:52.445120 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:52.445077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:53.706928 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:52:53.706872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:01.440194 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:01.440122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:53:03.942954 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:03.942912 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz"] Sep 11 18:53:03.943388 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:03.943170 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" podUID="a3dfdea1-ca52-417a-b051-d31dd980c720" containerName="maas-api" containerID="cri-o://0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba" gracePeriod=30 Sep 11 18:53:03.972535 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:03.972498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:04.074311 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.074266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:04.174145 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.174111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:04.194030 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.193963 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:53:04.203164 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.203136 2579 generic.go:356] "Generic (PLEG): container finished" podID="a3dfdea1-ca52-417a-b051-d31dd980c720" containerID="0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba" exitCode=0 Sep 11 18:53:04.203330 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.203178 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" Sep 11 18:53:04.203330 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.203179 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" event={"ID":"a3dfdea1-ca52-417a-b051-d31dd980c720","Type":"ContainerDied","Data":"0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba"} Sep 11 18:53:04.203330 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.203242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz" event={"ID":"a3dfdea1-ca52-417a-b051-d31dd980c720","Type":"ContainerDied","Data":"41ce3f22f32399c3b70e0062f6c261d0733fae86ecbb5e02ed7131bca5bbdfdc"} Sep 11 18:53:04.203330 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.203271 2579 scope.go:117] "RemoveContainer" containerID="0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba" Sep 11 18:53:04.212526 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.212452 2579 scope.go:117] "RemoveContainer" containerID="0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba" Sep 11 18:53:04.213071 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:53:04.212890 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba\": container with ID starting with 0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba not found: ID does not exist" containerID="0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba" Sep 11 18:53:04.213179 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.213047 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba"} err="failed to get container status \"0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba\": rpc error: code = NotFound desc = could not find container \"0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba\": container with ID starting with 0fa4f813892e26e48a175a348441f85574525ae2e393bc0e08eb49e6132ca4ba not found: ID does not exist" Sep 11 18:53:04.256724 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.256666 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dfdea1-ca52-417a-b051-d31dd980c720-openshift-service-ca\") pod \"a3dfdea1-ca52-417a-b051-d31dd980c720\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " Sep 11 18:53:04.256724 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.256725 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdf46\" (UniqueName: \"kubernetes.io/projected/a3dfdea1-ca52-417a-b051-d31dd980c720-kube-api-access-pdf46\") pod \"a3dfdea1-ca52-417a-b051-d31dd980c720\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " Sep 11 18:53:04.256941 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.256752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-metrics-certs\") pod \"a3dfdea1-ca52-417a-b051-d31dd980c720\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " Sep 11 18:53:04.256941 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.256788 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-maas-api-tls\") pod \"a3dfdea1-ca52-417a-b051-d31dd980c720\" (UID: \"a3dfdea1-ca52-417a-b051-d31dd980c720\") " Sep 11 18:53:04.257090 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.257065 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dfdea1-ca52-417a-b051-d31dd980c720-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3dfdea1-ca52-417a-b051-d31dd980c720" (UID: "a3dfdea1-ca52-417a-b051-d31dd980c720"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:53:04.258936 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.258905 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3dfdea1-ca52-417a-b051-d31dd980c720" (UID: "a3dfdea1-ca52-417a-b051-d31dd980c720"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:53:04.259075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.258943 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dfdea1-ca52-417a-b051-d31dd980c720-kube-api-access-pdf46" (OuterVolumeSpecName: "kube-api-access-pdf46") pod "a3dfdea1-ca52-417a-b051-d31dd980c720" (UID: "a3dfdea1-ca52-417a-b051-d31dd980c720"). InnerVolumeSpecName "kube-api-access-pdf46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:53:04.259075 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.258969 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a3dfdea1-ca52-417a-b051-d31dd980c720" (UID: "a3dfdea1-ca52-417a-b051-d31dd980c720"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:53:04.280599 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.280539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:04.358342 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.358300 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dfdea1-ca52-417a-b051-d31dd980c720-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:53:04.358342 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.358338 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdf46\" (UniqueName: \"kubernetes.io/projected/a3dfdea1-ca52-417a-b051-d31dd980c720-kube-api-access-pdf46\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:53:04.358563 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.358356 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:53:04.358563 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.358370 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dfdea1-ca52-417a-b051-d31dd980c720-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:53:04.527455 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.527423 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz"] Sep 11 18:53:04.533479 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.533451 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da4d13-54b798bd9-b9nqz"] Sep 11 18:53:04.571424 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.571393 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:04.681911 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:04.681873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:05.528452 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:05.528421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:06.229394 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:06.229359 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3dfdea1-ca52-417a-b051-d31dd980c720" path="/var/lib/kubelet/pods/a3dfdea1-ca52-417a-b051-d31dd980c720/volumes" Sep 11 18:53:06.301041 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:06.301011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:07.030817 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:07.030780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:09.655617 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:09.655584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:14.805665 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:14.805623 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:53:17.516641 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:17.516603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:36.710093 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:36.710053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:46.510148 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:46.510100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:46.869315 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:46.869216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:52.626243 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:52.626204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:58.478830 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:58.478785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:58.578883 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:58.578831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:58.780965 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:58.780915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:59.472805 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:53:59.472767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:00.148614 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:00.148574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:00.820122 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:00.820083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:01.634530 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:01.634488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:02.317437 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:02.317400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:08.618034 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:08.617977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:08.683155 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:08.683117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:09.791127 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:09.790858 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq"] Sep 11 18:54:09.791515 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:09.791386 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" podUID="a510cd4b-c710-4740-bc15-655db68cc15f" containerName="maas-api" containerID="cri-o://a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441" gracePeriod=30 Sep 11 18:54:09.834107 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:09.834069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:09.924749 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:09.924697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:10.023230 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.023193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:10.047875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.047808 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:54:10.123971 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.123937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:10.178149 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.178113 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmg4f\" (UniqueName: \"kubernetes.io/projected/a510cd4b-c710-4740-bc15-655db68cc15f-kube-api-access-fmg4f\") pod \"a510cd4b-c710-4740-bc15-655db68cc15f\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " Sep 11 18:54:10.178321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.178166 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510cd4b-c710-4740-bc15-655db68cc15f-openshift-service-ca\") pod \"a510cd4b-c710-4740-bc15-655db68cc15f\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " Sep 11 18:54:10.178321 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.178211 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-maas-api-tls\") pod \"a510cd4b-c710-4740-bc15-655db68cc15f\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " Sep 11 18:54:10.178440 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.178402 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-metrics-certs\") pod \"a510cd4b-c710-4740-bc15-655db68cc15f\" (UID: \"a510cd4b-c710-4740-bc15-655db68cc15f\") " Sep 11 18:54:10.178555 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.178532 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a510cd4b-c710-4740-bc15-655db68cc15f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a510cd4b-c710-4740-bc15-655db68cc15f" (UID: "a510cd4b-c710-4740-bc15-655db68cc15f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:54:10.178789 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.178769 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a510cd4b-c710-4740-bc15-655db68cc15f-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:54:10.180401 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.180369 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a510cd4b-c710-4740-bc15-655db68cc15f" (UID: "a510cd4b-c710-4740-bc15-655db68cc15f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:54:10.180517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.180416 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a510cd4b-c710-4740-bc15-655db68cc15f-kube-api-access-fmg4f" (OuterVolumeSpecName: "kube-api-access-fmg4f") pod "a510cd4b-c710-4740-bc15-655db68cc15f" (UID: "a510cd4b-c710-4740-bc15-655db68cc15f"). InnerVolumeSpecName "kube-api-access-fmg4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:54:10.180517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.180422 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a510cd4b-c710-4740-bc15-655db68cc15f" (UID: "a510cd4b-c710-4740-bc15-655db68cc15f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:54:10.279688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.279648 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmg4f\" (UniqueName: \"kubernetes.io/projected/a510cd4b-c710-4740-bc15-655db68cc15f-kube-api-access-fmg4f\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:54:10.279688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.279682 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:54:10.279688 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.279693 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a510cd4b-c710-4740-bc15-655db68cc15f-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:54:10.389246 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.389135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:10.494509 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.494469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:10.594849 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.594815 2579 generic.go:356] "Generic (PLEG): container finished" podID="a510cd4b-c710-4740-bc15-655db68cc15f" containerID="a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441" exitCode=0 Sep 11 18:54:10.595016 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.594867 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" Sep 11 18:54:10.595016 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.594900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" event={"ID":"a510cd4b-c710-4740-bc15-655db68cc15f","Type":"ContainerDied","Data":"a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441"} Sep 11 18:54:10.595016 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.594955 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq" event={"ID":"a510cd4b-c710-4740-bc15-655db68cc15f","Type":"ContainerDied","Data":"2cf4a700febc411313fc1a4be849bee8785f18bc13c1da42856f855ca8c5c5f6"} Sep 11 18:54:10.595016 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.594978 2579 scope.go:117] "RemoveContainer" containerID="a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441" Sep 11 18:54:10.604860 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.604845 2579 scope.go:117] "RemoveContainer" containerID="a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441" Sep 11 18:54:10.605157 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:54:10.605134 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441\": container with ID starting with a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441 not found: ID does not exist" containerID="a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441" Sep 11 18:54:10.605248 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.605167 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441"} err="failed to get container status \"a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441\": rpc error: code = NotFound desc = could not find container \"a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441\": container with ID starting with a200c5f61d0229892ddcbf022186dfb346c62ac2380f512e720bb3ab5f58d441 not found: ID does not exist" Sep 11 18:54:10.614997 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.614971 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq"] Sep 11 18:54:10.618850 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:10.618830 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ed2c8-7477747675-24zjq"] Sep 11 18:54:11.238502 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:11.238450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:11.956523 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:11.956473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:12.229008 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:12.228915 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a510cd4b-c710-4740-bc15-655db68cc15f" path="/var/lib/kubelet/pods/a510cd4b-c710-4740-bc15-655db68cc15f/volumes" Sep 11 18:54:13.250033 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:13.249995 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:54:16.401750 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:16.401696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:16.473186 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:16.473147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:16.695243 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:16.695155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:18.017325 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:18.017292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:18.713523 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:18.713489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:19.484364 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:19.484314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:27.724325 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:27.724217 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn"] Sep 11 18:54:27.724875 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:27.724490 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" podUID="1155c028-9680-4af3-a548-0b4bbca1d763" containerName="maas-api" containerID="cri-o://d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5" gracePeriod=30 Sep 11 18:54:27.759698 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:27.759667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:27.861476 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:27.861442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:27.953116 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:27.953078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:27.981069 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:27.981002 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:54:28.015162 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.015121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:28.039246 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.039212 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-maas-api-tls\") pod \"1155c028-9680-4af3-a548-0b4bbca1d763\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " Sep 11 18:54:28.039246 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.039248 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz76r\" (UniqueName: \"kubernetes.io/projected/1155c028-9680-4af3-a548-0b4bbca1d763-kube-api-access-mz76r\") pod \"1155c028-9680-4af3-a548-0b4bbca1d763\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " Sep 11 18:54:28.039440 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.039278 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1155c028-9680-4af3-a548-0b4bbca1d763-openshift-service-ca\") pod \"1155c028-9680-4af3-a548-0b4bbca1d763\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " Sep 11 18:54:28.039440 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.039416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-metrics-certs\") pod \"1155c028-9680-4af3-a548-0b4bbca1d763\" (UID: \"1155c028-9680-4af3-a548-0b4bbca1d763\") " Sep 11 18:54:28.039731 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.039687 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1155c028-9680-4af3-a548-0b4bbca1d763-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1155c028-9680-4af3-a548-0b4bbca1d763" (UID: "1155c028-9680-4af3-a548-0b4bbca1d763"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:54:28.041494 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.041460 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1155c028-9680-4af3-a548-0b4bbca1d763" (UID: "1155c028-9680-4af3-a548-0b4bbca1d763"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:54:28.041594 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.041486 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1155c028-9680-4af3-a548-0b4bbca1d763" (UID: "1155c028-9680-4af3-a548-0b4bbca1d763"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:54:28.041594 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.041527 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1155c028-9680-4af3-a548-0b4bbca1d763-kube-api-access-mz76r" (OuterVolumeSpecName: "kube-api-access-mz76r") pod "1155c028-9680-4af3-a548-0b4bbca1d763" (UID: "1155c028-9680-4af3-a548-0b4bbca1d763"). InnerVolumeSpecName "kube-api-access-mz76r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:54:28.140135 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.140099 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:54:28.140135 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.140132 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1155c028-9680-4af3-a548-0b4bbca1d763-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:54:28.140384 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.140145 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz76r\" (UniqueName: \"kubernetes.io/projected/1155c028-9680-4af3-a548-0b4bbca1d763-kube-api-access-mz76r\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:54:28.140384 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.140159 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1155c028-9680-4af3-a548-0b4bbca1d763-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:54:28.369811 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.369770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:28.464815 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.464764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:28.698769 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.698728 2579 generic.go:356] "Generic (PLEG): container finished" podID="1155c028-9680-4af3-a548-0b4bbca1d763" containerID="d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5" exitCode=0 Sep 11 18:54:28.698950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.698785 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" Sep 11 18:54:28.698950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.698824 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" event={"ID":"1155c028-9680-4af3-a548-0b4bbca1d763","Type":"ContainerDied","Data":"d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5"} Sep 11 18:54:28.698950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.698892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn" event={"ID":"1155c028-9680-4af3-a548-0b4bbca1d763","Type":"ContainerDied","Data":"7b663c98e5a5e48e08cb1f6e9cfef68ba28f80bef7a324110c2a4eaddb0ffc39"} Sep 11 18:54:28.698950 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.698926 2579 scope.go:117] "RemoveContainer" containerID="d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5" Sep 11 18:54:28.708517 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.708499 2579 scope.go:117] "RemoveContainer" containerID="d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5" Sep 11 18:54:28.708790 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:54:28.708770 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5\": container with ID starting with d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5 not found: ID does not exist" containerID="d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5" Sep 11 18:54:28.708877 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.708798 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5"} err="failed to get container status \"d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5\": rpc error: code = NotFound desc = could not find container \"d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5\": container with ID starting with d2071613e180f48dabdf8fb7fd17ba1b496c8ba0d21f18a1c63a4f17fb6096e5 not found: ID does not exist" Sep 11 18:54:28.719294 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.719272 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn"] Sep 11 18:54:28.723252 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:28.723229 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d90e71-79686b96bf-lp4xn"] Sep 11 18:54:29.118135 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:29.118097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:30.229558 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:30.229526 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1155c028-9680-4af3-a548-0b4bbca1d763" path="/var/lib/kubelet/pods/1155c028-9680-4af3-a548-0b4bbca1d763/volumes" Sep 11 18:54:30.844024 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:30.843991 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:54:34.366254 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:34.366217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:34.475125 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:34.475084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:34.810483 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:34.810452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:35.381814 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:35.381772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:36.269610 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:36.269572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:36.983456 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:36.983422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:45.588180 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:45.588136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:45.654456 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:45.654426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:45.756699 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:45.756660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:45.857259 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:45.857161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:46.163899 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:46.163793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:46.262384 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:46.262344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:48.153470 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:48.153436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:54:51.994292 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:51.994253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:52.058042 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:52.058005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:52.286407 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:52.286372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:57.548003 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:54:57.547969 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:55:14.275043 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:55:14.275001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:23.254330 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:55:23.254300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:32.471009 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:55:32.470979 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:55:32.488865 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:55:32.488841 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 18:56:07.690979 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:56:07.690897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:09.025944 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:56:09.025909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:18.086261 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:56:18.086219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:18.314963 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:56:18.314928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:18.340465 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:56:18.340380 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:56:19.883630 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:56:19.883593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:44.254118 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:56:44.254086 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:56:53.644664 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:56:53.644627 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:57:00.849436 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:00.849404 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:57:09.241238 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:09.241196 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:57:23.943329 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:23.943288 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:57:32.051051 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:32.050970 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:57:38.927573 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:38.927537 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq"] Sep 11 18:57:38.928040 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:38.927836 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" podUID="6a74684d-e528-4140-9ec8-bfe7fa37bf71" containerName="manager" containerID="cri-o://61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2" gracePeriod=10 Sep 11 18:57:39.170974 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.170946 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:57:39.182183 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.182110 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j29z\" (UniqueName: \"kubernetes.io/projected/6a74684d-e528-4140-9ec8-bfe7fa37bf71-kube-api-access-5j29z\") pod \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\" (UID: \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\") " Sep 11 18:57:39.182326 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.182235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a74684d-e528-4140-9ec8-bfe7fa37bf71-extensions-socket-volume\") pod \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\" (UID: \"6a74684d-e528-4140-9ec8-bfe7fa37bf71\") " Sep 11 18:57:39.182621 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.182594 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a74684d-e528-4140-9ec8-bfe7fa37bf71-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6a74684d-e528-4140-9ec8-bfe7fa37bf71" (UID: "6a74684d-e528-4140-9ec8-bfe7fa37bf71"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:57:39.184252 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.184229 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a74684d-e528-4140-9ec8-bfe7fa37bf71-kube-api-access-5j29z" (OuterVolumeSpecName: "kube-api-access-5j29z") pod "6a74684d-e528-4140-9ec8-bfe7fa37bf71" (UID: "6a74684d-e528-4140-9ec8-bfe7fa37bf71"). InnerVolumeSpecName "kube-api-access-5j29z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:57:39.283163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.283130 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a74684d-e528-4140-9ec8-bfe7fa37bf71-extensions-socket-volume\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:57:39.283163 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.283157 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j29z\" (UniqueName: \"kubernetes.io/projected/6a74684d-e528-4140-9ec8-bfe7fa37bf71-kube-api-access-5j29z\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 18:57:39.803602 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.803565 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a74684d-e528-4140-9ec8-bfe7fa37bf71" containerID="61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2" exitCode=0 Sep 11 18:57:39.803868 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.803613 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" Sep 11 18:57:39.803868 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.803651 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" event={"ID":"6a74684d-e528-4140-9ec8-bfe7fa37bf71","Type":"ContainerDied","Data":"61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2"} Sep 11 18:57:39.803868 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.803726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq" event={"ID":"6a74684d-e528-4140-9ec8-bfe7fa37bf71","Type":"ContainerDied","Data":"c6c0322e4ffd9adba2588203bc69c9b49531a1ffae51015968a9ff36ad88f0c9"} Sep 11 18:57:39.803868 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.803751 2579 scope.go:117] "RemoveContainer" containerID="61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2" Sep 11 18:57:39.813871 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.813852 2579 scope.go:117] "RemoveContainer" containerID="61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2" Sep 11 18:57:39.814140 ip-10-0-142-253 kubenswrapper[2579]: E0911 18:57:39.814124 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2\": container with ID starting with 61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2 not found: ID does not exist" containerID="61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2" Sep 11 18:57:39.814185 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.814148 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2"} err="failed to get container status \"61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2\": rpc error: code = NotFound desc = could not find container \"61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2\": container with ID starting with 61b519690f5761bc4750851fabca7bf56515ab88d90843f60951cfbf2a78b3f2 not found: ID does not exist" Sep 11 18:57:39.825406 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.825382 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq"] Sep 11 18:57:39.831144 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:39.831119 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z4lfq"] Sep 11 18:57:40.229373 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:40.229286 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a74684d-e528-4140-9ec8-bfe7fa37bf71" path="/var/lib/kubelet/pods/6a74684d-e528-4140-9ec8-bfe7fa37bf71/volumes" Sep 11 18:57:53.479491 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:57:53.479452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:58:39.603438 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.603395 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b"] Sep 11 18:58:39.604000 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.603980 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a510cd4b-c710-4740-bc15-655db68cc15f" containerName="maas-api" Sep 11 18:58:39.604086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604002 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a510cd4b-c710-4740-bc15-655db68cc15f" containerName="maas-api" Sep 11 18:58:39.604086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604014 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1155c028-9680-4af3-a548-0b4bbca1d763" containerName="maas-api" Sep 11 18:58:39.604086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604022 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1155c028-9680-4af3-a548-0b4bbca1d763" containerName="maas-api" Sep 11 18:58:39.604086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604034 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" Sep 11 18:58:39.604086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604043 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" Sep 11 18:58:39.604086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604052 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864fe228-04ee-46ec-87b4-6b497d88fab1" containerName="maas-api" Sep 11 18:58:39.604086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604061 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="864fe228-04ee-46ec-87b4-6b497d88fab1" containerName="maas-api" Sep 11 18:58:39.604086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604078 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a74684d-e528-4140-9ec8-bfe7fa37bf71" containerName="manager" Sep 11 18:58:39.604086 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604087 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a74684d-e528-4140-9ec8-bfe7fa37bf71" containerName="manager" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604103 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3dfdea1-ca52-417a-b051-d31dd980c720" containerName="maas-api" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604112 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3dfdea1-ca52-417a-b051-d31dd980c720" containerName="maas-api" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604128 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604136 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604252 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a510cd4b-c710-4740-bc15-655db68cc15f" containerName="maas-api" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604265 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="864fe228-04ee-46ec-87b4-6b497d88fab1" containerName="maas-api" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604273 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604282 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a74684d-e528-4140-9ec8-bfe7fa37bf71" containerName="manager" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604289 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3dfdea1-ca52-417a-b051-d31dd980c720" containerName="maas-api" Sep 11 18:58:39.604528 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.604301 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1155c028-9680-4af3-a548-0b4bbca1d763" containerName="maas-api" Sep 11 18:58:39.608629 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.608607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:58:39.611200 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.611172 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rzr4j\"" Sep 11 18:58:39.619289 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.619267 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b"] Sep 11 18:58:39.726754 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.726699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4baa13bc-b39d-49f2-8e27-210f7cbf1d5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rcj4b\" (UID: \"4baa13bc-b39d-49f2-8e27-210f7cbf1d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:58:39.726935 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.726809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxqc\" (UniqueName: \"kubernetes.io/projected/4baa13bc-b39d-49f2-8e27-210f7cbf1d5b-kube-api-access-qbxqc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rcj4b\" (UID: \"4baa13bc-b39d-49f2-8e27-210f7cbf1d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:58:39.828287 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.828253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxqc\" (UniqueName: \"kubernetes.io/projected/4baa13bc-b39d-49f2-8e27-210f7cbf1d5b-kube-api-access-qbxqc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rcj4b\" (UID: \"4baa13bc-b39d-49f2-8e27-210f7cbf1d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:58:39.828477 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.828343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4baa13bc-b39d-49f2-8e27-210f7cbf1d5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rcj4b\" (UID: \"4baa13bc-b39d-49f2-8e27-210f7cbf1d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:58:39.828794 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.828770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4baa13bc-b39d-49f2-8e27-210f7cbf1d5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rcj4b\" (UID: \"4baa13bc-b39d-49f2-8e27-210f7cbf1d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:58:39.839630 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.839607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxqc\" (UniqueName: \"kubernetes.io/projected/4baa13bc-b39d-49f2-8e27-210f7cbf1d5b-kube-api-access-qbxqc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rcj4b\" (UID: \"4baa13bc-b39d-49f2-8e27-210f7cbf1d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:58:39.919034 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:39.918959 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:58:40.197724 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:40.197682 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b"] Sep 11 18:58:40.199514 ip-10-0-142-253 kubenswrapper[2579]: W0911 18:58:40.199482 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4baa13bc_b39d_49f2_8e27_210f7cbf1d5b.slice/crio-52c1b5f107841cf591a674d0b9fed37727919df0d908e003644fce85bd35ab2d WatchSource:0}: Error finding container 52c1b5f107841cf591a674d0b9fed37727919df0d908e003644fce85bd35ab2d: Status 404 returned error can't find the container with id 52c1b5f107841cf591a674d0b9fed37727919df0d908e003644fce85bd35ab2d Sep 11 18:58:41.152795 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:41.152761 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" event={"ID":"4baa13bc-b39d-49f2-8e27-210f7cbf1d5b","Type":"ContainerStarted","Data":"1bd95e60c5b930535ab462b1d7652a7859b0328799b95baa17c542d1eda07fb7"} Sep 11 18:58:41.153278 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:41.152812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" event={"ID":"4baa13bc-b39d-49f2-8e27-210f7cbf1d5b","Type":"ContainerStarted","Data":"52c1b5f107841cf591a674d0b9fed37727919df0d908e003644fce85bd35ab2d"} Sep 11 18:58:41.153278 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:41.152856 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:58:41.177461 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:41.177410 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" podStartSLOduration=2.177394314 podStartE2EDuration="2.177394314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:58:41.174769356 +0000 UTC m=+2589.401445707" watchObservedRunningTime="2026-09-11 18:58:41.177394314 +0000 UTC m=+2589.404070694" Sep 11 18:58:52.160864 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:58:52.160829 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rcj4b" Sep 11 18:59:00.346722 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:00.346620 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:59:07.241009 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:07.240967 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:59:14.643028 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:14.642987 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:59:20.072970 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:20.072930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:59:22.746771 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:22.746731 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:59:37.974545 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:37.974509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:59:40.753641 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:40.753604 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 18:59:54.692839 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:54.692805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:59:56.060588 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:56.060553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:59:56.584102 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:56.584058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 18:59:56.747327 ip-10-0-142-253 kubenswrapper[2579]: I0911 18:59:56.747291 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 19:00:05.940472 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:05.940429 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 19:00:13.145275 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:13.145239 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 19:00:19.941522 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:19.941486 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 19:00:20.200659 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.200565 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-ppxh7"] Sep 11 19:00:20.201146 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.201125 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9262010-ced6-42e0-842c-3a266af78379" containerName="istio-proxy" Sep 11 19:00:20.204310 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.204289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.207904 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.207880 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 19:00:20.208037 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.207880 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 19:00:20.208037 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.207965 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:00:20.208037 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.207987 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 19:00:20.208037 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.207921 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l7zn9\"" Sep 11 19:00:20.216569 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.216547 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-ppxh7"] Sep 11 19:00:20.216957 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.216931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-openshift-service-ca\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.217059 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.217028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-metrics-certs\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.217131 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.217103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvrww\" (UniqueName: \"kubernetes.io/projected/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-kube-api-access-cvrww\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.217188 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.217155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-maas-api-tls\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.218904 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.218881 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg"] Sep 11 19:00:20.228016 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.227991 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.243857 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.243831 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg"] Sep 11 19:00:20.318295 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.318258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvrww\" (UniqueName: \"kubernetes.io/projected/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-kube-api-access-cvrww\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.318295 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.318304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-maas-api-tls\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.318511 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.318331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-openshift-service-ca\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.318511 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.318354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98jt\" (UniqueName: \"kubernetes.io/projected/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-kube-api-access-b98jt\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.318511 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.318373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-openshift-service-ca\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.318511 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.318402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-maas-api-tls\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.318511 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.318463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-metrics-certs\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.318676 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.318515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-metrics-certs\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.319053 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.319024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-openshift-service-ca\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.320944 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.320922 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-metrics-certs\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.321062 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.321041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-maas-api-tls\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.341448 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.341418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvrww\" (UniqueName: \"kubernetes.io/projected/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-kube-api-access-cvrww\") pod \"maas-api-976f486c4-ppxh7\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.419420 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.419385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-openshift-service-ca\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.419598 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.419434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98jt\" (UniqueName: \"kubernetes.io/projected/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-kube-api-access-b98jt\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.419598 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.419481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-maas-api-tls\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.419598 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.419501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-metrics-certs\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.420057 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.420030 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-openshift-service-ca\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.421856 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.421832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-metrics-certs\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.421956 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.421935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-maas-api-tls\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.426939 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.426921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98jt\" (UniqueName: \"kubernetes.io/projected/9128aa24-3b33-48ad-8d27-b6760bf1d6a1-kube-api-access-b98jt\") pod \"maas-api-765c4fc4c8-xshjg\" (UID: \"9128aa24-3b33-48ad-8d27-b6760bf1d6a1\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.518001 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.517976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:20.544131 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.544100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:20.795478 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.795451 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-ppxh7"] Sep 11 19:00:20.797506 ip-10-0-142-253 kubenswrapper[2579]: W0911 19:00:20.797471 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595d5cdd_7ae9_4092_b0e0_0f7e92ff3515.slice/crio-2a0ea45b2d9cbda88aad74064cfd1145ad4b8f080585f22dd9ab012c50ca501a WatchSource:0}: Error finding container 2a0ea45b2d9cbda88aad74064cfd1145ad4b8f080585f22dd9ab012c50ca501a: Status 404 returned error can't find the container with id 2a0ea45b2d9cbda88aad74064cfd1145ad4b8f080585f22dd9ab012c50ca501a Sep 11 19:00:20.799129 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.799108 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:00:20.826372 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:20.826351 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg"] Sep 11 19:00:20.828372 ip-10-0-142-253 kubenswrapper[2579]: W0911 19:00:20.828348 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9128aa24_3b33_48ad_8d27_b6760bf1d6a1.slice/crio-597ccc8725308789cbf37ba2e1b95db1f1eabb0ab17780416ae7d896d70c5bbd WatchSource:0}: Error finding container 597ccc8725308789cbf37ba2e1b95db1f1eabb0ab17780416ae7d896d70c5bbd: Status 404 returned error can't find the container with id 597ccc8725308789cbf37ba2e1b95db1f1eabb0ab17780416ae7d896d70c5bbd Sep 11 19:00:21.732803 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:21.732680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" event={"ID":"9128aa24-3b33-48ad-8d27-b6760bf1d6a1","Type":"ContainerStarted","Data":"f57b4d07f810224f34d41cf9d6e24c645ea908ea553941fc5fcb6d2101b57513"} Sep 11 19:00:21.732803 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:21.732767 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:21.732803 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:21.732790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" event={"ID":"9128aa24-3b33-48ad-8d27-b6760bf1d6a1","Type":"ContainerStarted","Data":"597ccc8725308789cbf37ba2e1b95db1f1eabb0ab17780416ae7d896d70c5bbd"} Sep 11 19:00:21.734250 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:21.734219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" event={"ID":"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515","Type":"ContainerStarted","Data":"f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790"} Sep 11 19:00:21.734378 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:21.734269 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" event={"ID":"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515","Type":"ContainerStarted","Data":"2a0ea45b2d9cbda88aad74064cfd1145ad4b8f080585f22dd9ab012c50ca501a"} Sep 11 19:00:21.734378 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:21.734290 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:21.750721 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:21.750662 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" podStartSLOduration=1.280475701 podStartE2EDuration="1.750648276s" totalImagesPullingTime="470.172575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:21.749695234 +0000 UTC m=+2689.976371594" watchObservedRunningTime="2026-09-11 19:00:21.750648276 +0000 UTC m=+2689.977324616" Sep 11 19:00:21.771060 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:21.771020 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" podStartSLOduration=1.240393206 podStartE2EDuration="1.771007168s" totalImagesPullingTime="530.613962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:21.769042328 +0000 UTC m=+2689.995718688" watchObservedRunningTime="2026-09-11 19:00:21.771007168 +0000 UTC m=+2689.997683530" Sep 11 19:00:27.645109 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:27.645026 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 19:00:27.744778 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:27.744741 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-xshjg" Sep 11 19:00:27.745066 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:27.745049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:27.810242 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:27.810209 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-ppxh7"] Sep 11 19:00:27.810485 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:27.810448 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" podUID="595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" containerName="maas-api" containerID="cri-o://f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790" gracePeriod=30 Sep 11 19:00:28.063974 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.063945 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:28.085950 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.085923 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvrww\" (UniqueName: \"kubernetes.io/projected/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-kube-api-access-cvrww\") pod \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " Sep 11 19:00:28.086101 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.086021 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-metrics-certs\") pod \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " Sep 11 19:00:28.086101 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.086069 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-maas-api-tls\") pod \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " Sep 11 19:00:28.086278 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.086252 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-openshift-service-ca\") pod \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\" (UID: \"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515\") " Sep 11 19:00:28.086681 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.086635 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" (UID: "595d5cdd-7ae9-4092-b0e0-0f7e92ff3515"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:00:28.089274 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.089227 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" (UID: "595d5cdd-7ae9-4092-b0e0-0f7e92ff3515"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:00:28.089497 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.089295 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" (UID: "595d5cdd-7ae9-4092-b0e0-0f7e92ff3515"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:00:28.089497 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.089366 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-kube-api-access-cvrww" (OuterVolumeSpecName: "kube-api-access-cvrww") pod "595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" (UID: "595d5cdd-7ae9-4092-b0e0-0f7e92ff3515"). InnerVolumeSpecName "kube-api-access-cvrww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:00:28.187225 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.187128 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvrww\" (UniqueName: \"kubernetes.io/projected/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-kube-api-access-cvrww\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 19:00:28.187225 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.187157 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-metrics-certs\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 19:00:28.187225 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.187167 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-maas-api-tls\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 19:00:28.187225 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.187175 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515-openshift-service-ca\") on node \"ip-10-0-142-253.ec2.internal\" DevicePath \"\"" Sep 11 19:00:28.774031 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.773998 2579 generic.go:356] "Generic (PLEG): container finished" podID="595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" containerID="f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790" exitCode=0 Sep 11 19:00:28.774485 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.774043 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" Sep 11 19:00:28.774485 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.774058 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" event={"ID":"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515","Type":"ContainerDied","Data":"f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790"} Sep 11 19:00:28.774485 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.774100 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976f486c4-ppxh7" event={"ID":"595d5cdd-7ae9-4092-b0e0-0f7e92ff3515","Type":"ContainerDied","Data":"2a0ea45b2d9cbda88aad74064cfd1145ad4b8f080585f22dd9ab012c50ca501a"} Sep 11 19:00:28.774485 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.774121 2579 scope.go:117] "RemoveContainer" containerID="f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790" Sep 11 19:00:28.783426 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.783406 2579 scope.go:117] "RemoveContainer" containerID="f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790" Sep 11 19:00:28.783722 ip-10-0-142-253 kubenswrapper[2579]: E0911 19:00:28.783680 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790\": container with ID starting with f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790 not found: ID does not exist" containerID="f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790" Sep 11 19:00:28.783810 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.783737 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790"} err="failed to get container status \"f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790\": rpc error: code = NotFound desc = could not find container \"f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790\": container with ID starting with f4da94d52c3c9f1c6776ec785912162807a845c8e5fc0d165e2e21321c45e790 not found: ID does not exist" Sep 11 19:00:28.792222 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.792197 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-ppxh7"] Sep 11 19:00:28.798482 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:28.798461 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-ppxh7"] Sep 11 19:00:30.228994 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:30.228961 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" path="/var/lib/kubelet/pods/595d5cdd-7ae9-4092-b0e0-0f7e92ff3515/volumes" Sep 11 19:00:32.515239 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:32.515211 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 19:00:32.528656 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:32.528634 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d4sfd_377ee3bf-f751-4c9c-9552-eb50fa46d6d5/ovn-acl-logging/0.log" Sep 11 19:00:34.439164 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:34.439122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 19:00:40.138763 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:40.138727 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 19:00:47.344886 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:47.344854 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-qd92k"] Sep 11 19:00:50.154577 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:50.154504 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748dbd7f6-bz497_1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa/authorino/0.log" Sep 11 19:00:55.555582 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:55.555550 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xdcc4_83530f82-2858-426f-bcee-886bec79f531/manager/0.log" Sep 11 19:00:55.673231 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:55.673202 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mptwq_1e545f22-c74a-4d15-a94f-884b2c4e5b24/manager/0.log" Sep 11 19:00:55.916548 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:55.916475 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lfglc_6f118ba9-bb39-4e13-8fd1-2b4ac360259c/server/0.log" Sep 11 19:00:56.431054 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:56.431020 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b7b6b756-mwrq6_9dd225af-1468-4cbd-a532-9d0e0fb1a317/copy-manifests/0.log" Sep 11 19:00:56.452145 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:56.452114 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b7b6b756-mwrq6_9dd225af-1468-4cbd-a532-9d0e0fb1a317/manager/0.log" Sep 11 19:00:56.910923 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:56.910884 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-765c4fc4c8-xshjg_9128aa24-3b33-48ad-8d27-b6760bf1d6a1/maas-api/0.log" Sep 11 19:00:58.116415 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:58.116384 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748dbd7f6-bz497_1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa/authorino/0.log" Sep 11 19:00:58.240570 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:58.240541 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-msdcq_c7eea2f1-60ef-488d-96a0-bb43497e6756/manager/0.log" Sep 11 19:00:58.352879 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:58.352852 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-bh7nx_e841beac-389b-4feb-83f0-f365d3e1f162/manager/0.log" Sep 11 19:00:58.464912 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:58.464841 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9xms2_e1674050-c7ab-4053-a394-fbf98776e0ac/kuadrant-console-plugin/0.log" Sep 11 19:00:58.587279 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:58.587253 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-rcj4b_4baa13bc-b39d-49f2-8e27-210f7cbf1d5b/manager/0.log" Sep 11 19:00:58.703968 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:58.703943 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-qd92k_eab82b8c-cc58-4fb5-8a98-1db16127d5a3/limitador/0.log" Sep 11 19:00:58.814300 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:58.814270 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-4jqrm_bb9ec434-bb32-40ca-b3c0-7b3be1ad352b/manager/0.log" Sep 11 19:00:59.391515 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:00:59.391492 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58fb58cc-bdr74_75441e84-9578-4566-9ed8-1538aee9d55b/kube-auth-proxy/0.log" Sep 11 19:01:00.349715 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:00.349684 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-fq2bl_74da949f-aa62-4be3-89f4-3996004163bc/main/0.log" Sep 11 19:01:00.688495 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:00.688391 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6mqft_ff43b734-61c2-4d36-b1d2-c9343860a4ce/main/0.log" Sep 11 19:01:01.049036 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:01.049010 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pjglq_b08a45f6-a5c5-4b06-ad5c-cf19f2d01683/main/0.log" Sep 11 19:01:06.248302 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:06.248270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:06.414546 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:06.414509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:10.077041 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:10.076996 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jphcv_f4d8133f-e8a5-42a2-89e4-8c24af6d2ac3/global-pull-secret-syncer/0.log" Sep 11 19:01:10.199988 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:10.199898 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cgv47_34e112ec-e721-4ded-a94d-f259b4e2e2b8/konnectivity-agent/0.log" Sep 11 19:01:10.370631 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:10.370599 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-253.ec2.internal_6b275f54185b5d5dddefd04d62d766a8/haproxy/0.log" Sep 11 19:01:14.609375 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:14.609339 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748dbd7f6-bz497_1fb8f7e7-5cc9-4f6e-b50f-55c03c9b47fa/authorino/0.log" Sep 11 19:01:14.645175 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:14.645144 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-msdcq_c7eea2f1-60ef-488d-96a0-bb43497e6756/manager/0.log" Sep 11 19:01:14.687299 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:14.687272 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-bh7nx_e841beac-389b-4feb-83f0-f365d3e1f162/manager/0.log" Sep 11 19:01:14.714576 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:14.714546 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9xms2_e1674050-c7ab-4053-a394-fbf98776e0ac/kuadrant-console-plugin/0.log" Sep 11 19:01:14.757699 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:14.757665 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-rcj4b_4baa13bc-b39d-49f2-8e27-210f7cbf1d5b/manager/0.log" Sep 11 19:01:14.793565 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:14.793516 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-qd92k_eab82b8c-cc58-4fb5-8a98-1db16127d5a3/limitador/0.log" Sep 11 19:01:14.858585 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:14.858553 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-4jqrm_bb9ec434-bb32-40ca-b3c0-7b3be1ad352b/manager/0.log" Sep 11 19:01:16.365157 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.365131 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4799c69c-32cc-4d73-8c4f-28627aa7981a/alertmanager/0.log" Sep 11 19:01:16.403457 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.403430 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4799c69c-32cc-4d73-8c4f-28627aa7981a/config-reloader/0.log" Sep 11 19:01:16.435280 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.435250 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4799c69c-32cc-4d73-8c4f-28627aa7981a/kube-rbac-proxy-web/0.log" Sep 11 19:01:16.469561 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.469536 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4799c69c-32cc-4d73-8c4f-28627aa7981a/kube-rbac-proxy/0.log" Sep 11 19:01:16.513677 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.513639 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4799c69c-32cc-4d73-8c4f-28627aa7981a/kube-rbac-proxy-metric/0.log" Sep 11 19:01:16.544687 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.544663 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4799c69c-32cc-4d73-8c4f-28627aa7981a/prom-label-proxy/0.log" Sep 11 19:01:16.568867 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.568823 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4799c69c-32cc-4d73-8c4f-28627aa7981a/init-config-reloader/0.log" Sep 11 19:01:16.616201 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.616119 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-jf6v4_bab5ef01-e412-4447-a555-2913d63a81c6/cluster-monitoring-operator/0.log" Sep 11 19:01:16.885429 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.885349 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gj24r_0feb5979-7e5c-4f24-8446-34510ef61780/node-exporter/0.log" Sep 11 19:01:16.916192 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.916167 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gj24r_0feb5979-7e5c-4f24-8446-34510ef61780/kube-rbac-proxy/0.log" Sep 11 19:01:16.942605 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:16.942583 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gj24r_0feb5979-7e5c-4f24-8446-34510ef61780/init-textfile/0.log" Sep 11 19:01:17.890222 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:17.890176 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-6svkg_37608dbc-bbcf-484e-927a-457ccafcf4c8/prometheus-operator-admission-webhook/0.log" Sep 11 19:01:17.965305 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:17.965261 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-794746b5b4-4n628_7081f1b8-b49f-43f0-a2f2-4abdc52f75cf/telemeter-client/0.log" Sep 11 19:01:17.996681 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:17.996655 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-794746b5b4-4n628_7081f1b8-b49f-43f0-a2f2-4abdc52f75cf/reload/0.log" Sep 11 19:01:18.027789 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:18.027766 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-794746b5b4-4n628_7081f1b8-b49f-43f0-a2f2-4abdc52f75cf/kube-rbac-proxy/0.log" Sep 11 19:01:19.395630 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:19.395605 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-bf96p_d73d283b-2bdf-473f-9833-67b7c2ca8a62/networking-console-plugin/0.log" Sep 11 19:01:20.119973 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.119945 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz"] Sep 11 19:01:20.120364 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.120351 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" containerName="maas-api" Sep 11 19:01:20.120417 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.120366 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" containerName="maas-api" Sep 11 19:01:20.120460 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.120434 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="595d5cdd-7ae9-4092-b0e0-0f7e92ff3515" containerName="maas-api" Sep 11 19:01:20.123576 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.123558 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.126097 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.126074 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5r55n\"/\"kube-root-ca.crt\"" Sep 11 19:01:20.127000 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.126983 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5r55n\"/\"default-dockercfg-rblx9\"" Sep 11 19:01:20.127059 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.126983 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5r55n\"/\"openshift-service-ca.crt\"" Sep 11 19:01:20.133329 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.133306 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz"] Sep 11 19:01:20.152759 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.152734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-lib-modules\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.152894 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.152764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzsq\" (UniqueName: \"kubernetes.io/projected/9f15f5d9-3606-4d3f-a557-fadc83a07392-kube-api-access-6dzsq\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.152894 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.152785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-podres\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.152894 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.152878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-proc\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.153020 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.152927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-sys\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.253424 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.253392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-lib-modules\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.253424 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.253427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzsq\" (UniqueName: \"kubernetes.io/projected/9f15f5d9-3606-4d3f-a557-fadc83a07392-kube-api-access-6dzsq\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.253655 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.253448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-podres\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.253655 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.253550 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-podres\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.253655 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.253575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-lib-modules\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.253655 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.253632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-proc\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.253873 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.253724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-sys\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.253873 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.253754 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-proc\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.253873 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.253831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f15f5d9-3606-4d3f-a557-fadc83a07392-sys\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.270476 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.270454 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzsq\" (UniqueName: \"kubernetes.io/projected/9f15f5d9-3606-4d3f-a557-fadc83a07392-kube-api-access-6dzsq\") pod \"perf-node-gather-daemonset-655mz\" (UID: \"9f15f5d9-3606-4d3f-a557-fadc83a07392\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.434200 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.434113 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:20.708551 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.708524 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz"] Sep 11 19:01:20.710611 ip-10-0-142-253 kubenswrapper[2579]: W0911 19:01:20.710578 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f15f5d9_3606_4d3f_a557_fadc83a07392.slice/crio-6a3e216b077e4b041f12d4d93f3eddb62b1e70b5764743de564056ea696bb763 WatchSource:0}: Error finding container 6a3e216b077e4b041f12d4d93f3eddb62b1e70b5764743de564056ea696bb763: Status 404 returned error can't find the container with id 6a3e216b077e4b041f12d4d93f3eddb62b1e70b5764743de564056ea696bb763 Sep 11 19:01:20.740783 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.740753 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-868787cd89-9smjn_3bf8a114-aaad-4322-b7dd-0bdd82d91a1f/console/0.log" Sep 11 19:01:20.793295 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:20.793274 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-2khhb_394a1856-2042-4876-8df1-7701baa4de0f/download-server/0.log" Sep 11 19:01:21.081155 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:21.081115 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" event={"ID":"9f15f5d9-3606-4d3f-a557-fadc83a07392","Type":"ContainerStarted","Data":"51a36e616b7c866a5a5902a315f319604f2a996cf7981d4215f59db67713f8f2"} Sep 11 19:01:21.081341 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:21.081167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" event={"ID":"9f15f5d9-3606-4d3f-a557-fadc83a07392","Type":"ContainerStarted","Data":"6a3e216b077e4b041f12d4d93f3eddb62b1e70b5764743de564056ea696bb763"} Sep 11 19:01:21.081341 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:21.081223 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:21.097965 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:21.097920 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" podStartSLOduration=1.09790827 podStartE2EDuration="1.09790827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:01:21.096151029 +0000 UTC m=+2749.322827387" watchObservedRunningTime="2026-09-11 19:01:21.09790827 +0000 UTC m=+2749.324584610" Sep 11 19:01:22.592769 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:22.592740 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bhn27_7c2e0dbc-ad02-49f5-8604-4a45cfba2237/dns/0.log" Sep 11 19:01:22.616841 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:22.616816 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bhn27_7c2e0dbc-ad02-49f5-8604-4a45cfba2237/kube-rbac-proxy/0.log" Sep 11 19:01:22.713510 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:22.713472 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mn77k_27a6d135-2518-45ee-b801-709400c2a1a0/dns-node-resolver/0.log" Sep 11 19:01:23.506368 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:23.506335 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkmbt_cbaaf296-872b-4b25-b8da-17a8f77ad1d3/node-ca/0.log" Sep 11 19:01:25.029425 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:25.029393 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58fb58cc-bdr74_75441e84-9578-4566-9ed8-1538aee9d55b/kube-auth-proxy/0.log" Sep 11 19:01:26.099720 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:26.099680 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4f5s2_0524f5b1-df3e-4288-8864-709be1f0f4e4/serve-healthcheck-canary/0.log" Sep 11 19:01:26.780464 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:26.780429 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pjtwp_b3504eb3-b69b-49e0-a793-2ede1f938615/kube-rbac-proxy/0.log" Sep 11 19:01:26.807307 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:26.807275 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pjtwp_b3504eb3-b69b-49e0-a793-2ede1f938615/exporter/0.log" Sep 11 19:01:26.829458 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:26.829435 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pjtwp_b3504eb3-b69b-49e0-a793-2ede1f938615/extractor/0.log" Sep 11 19:01:27.097338 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:27.097268 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-655mz" Sep 11 19:01:29.046557 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:29.046517 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xdcc4_83530f82-2858-426f-bcee-886bec79f531/manager/0.log" Sep 11 19:01:29.294763 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:29.294729 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mptwq_1e545f22-c74a-4d15-a94f-884b2c4e5b24/manager/0.log" Sep 11 19:01:29.397998 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:29.397972 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lfglc_6f118ba9-bb39-4e13-8fd1-2b4ac360259c/server/0.log" Sep 11 19:01:29.782606 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:29.782576 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b7b6b756-mwrq6_9dd225af-1468-4cbd-a532-9d0e0fb1a317/manager/0.log" Sep 11 19:01:29.805875 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:29.805846 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b7b6b756-mwrq6_9dd225af-1468-4cbd-a532-9d0e0fb1a317/copy-manifests/0.log" Sep 11 19:01:31.528244 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:31.528218 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58fb58cc-bdr74_75441e84-9578-4566-9ed8-1538aee9d55b/kube-auth-proxy/0.log" Sep 11 19:01:33.672403 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:33.672366 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58fb58cc-bdr74_75441e84-9578-4566-9ed8-1538aee9d55b/kube-auth-proxy/0.log" Sep 11 19:01:36.926416 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:36.926381 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-2xhfd_8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84/kube-storage-version-migrator-operator/1.log" Sep 11 19:01:36.927258 ip-10-0-142-253 kubenswrapper[2579]: I0911 19:01:36.927237 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-2xhfd_8efe5c71-76f2-4ff3-9793-b8d4dcdb7d84/kube-storage-version-migrator-operator/0.log"